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May   19,   1976 


Honorable  Abe  Rlbicoff 

Chairman 

Committee  on  Government  Operations 

United  States  Senate 

Washington,  D.C.  20510 

Dear  Mr.  Chairman: 

On  behalf  of  the  Subcommittee  on  Oversight  Procedures, 
I  hereby  submit  a  report,  "Legislative  Oversight  and 
Program  Evaluation:   Proceedings  of  a  Seminar  Sponsored 
by  the  Congressional  Research  Service,  May,  1975." 

This  report  was  compiled  by  the  Congressional  Research 
Service,  Library  of  Congress,  at  my  request.   It  contains 
papers  and  other  materials  prepared  in  connection  with  a 
seminar  sponsored  by  CRS  in  May  of  1975  at  the  initiation 
of  Senators  Lee  Metcalf  and  Bill  Brock. 

The  seminar  was  designed  to  acquaint  congressional  staff 
with  the  various  techniques  of  oversight  and  the  resources 
available  to  them.   Special  emphasis  was  given  to  program 
review  and  evaluation. 

The  Subcommittee  on  Oversight  Procedures  has  a 
responsibility  to  develop  more  effective  methods  of  overseeing 
the  Executive  branch  of  the  government.   The  proceedings  of 
this  seminar  should  contribute  to  a  better  understanding  of 
oversight  methods  on  the  part  of  congressional  staff  and, 
consequently,  should  lead  to  more  effective  oversight. 

On  behalf  of  the  Subcommittee,  I  respectfully  request 
that  this  report  be  printed  for  the  use  of  the  Government 
Operations  Committee  in  order  to  make  the  materials  available 
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The  Honorable 

Sam  Nunn 

Chairman,  Subcommittee  on  Oversight 

Procedures 
Committee  on  Government  Operations 
United  States  Senate 
Carroll  Arms  Hotel  --  Room  502 
Washington,  D.  C.  20510 

Dear  Mr.  Chairman: 

I  am  pleased  to  transmit  this  material  which  you  requested  us 
to  prepare  for  possible  publication  as  a  committee  print.  The  document, 
"Legislative  Oversight  and  Evaluation:  Proceedings  of  a  Seminar  Spon- 
sored by  the  Congressional  Research  Service,  May  1975,"  contains  the 
proceedings  of  the  seminar  series  held  on  May  5,  12,  and  19,  1975. 

As  you  will  recall,  the  series  was  requested  by  Senator  Lee 
Metcalf,  Chairman  of  the  Government  Operations  Subcommittee  on  Reports, 
Accounting,  and  Management  and  Vice  Chairman  of  the  Joint  Committee  on 
Congressional  Operations,  and  by  Senator  Bill  Brock,  Ranking  Minority 
Member  of  the  Subcommittee  on  Reports,  Accounting,  and  Management  of 
the  Government  Operations  Committee. 

Following  the  seminar,  we  received  your  request  to  prepare 
the  proceedings  for  possible  publication.  You  noted  that  "...the 
Subcommittee  is  planning  to  publish  a  series  of  documents  and  prints 
on  oversight  and  evaluation  for  legislative  staff  orientation  and 
reference  purposes.  The  publication  of  the  seminar  series  proceedings 
in  committee  print  form  would  be  a  valuable  contribution  to  the  Sub- 
committee's program  of  improving  information  and  knowledge  about  leg- 
islative oversight  and  evaluation." 

As  requested  by  your  staff  we  have  prepared  a  brief  summary 
of  the  main  themes  of  the  seminar  series.  A  more  detailed  summary  of 
each  of  the  three  sessions  precedes  the  actual  papeis  and  discussion 
for  that  session.  We  have  also  included  appendix  materials  that 
speakers  distributed  at  the  meetings  to  illustrate  their  oversight 
and  evaluation  activities. 

We  should  like  to  acknowledge  the  participation  of  the  many 
persons  who  cooperated  with  us  in  presenting  the  seminar,  delivering 
papers,  and  preparing  the  proceedings.  The  Oversight  Procedures  Sub- 
committee staff  and  the  staff  of  the  Subcommittee  on  Reports,  Accounting 

(ix) 


and  Management  deserve  special  thanks.  We  should  also  like  to  acknow- 
ledge the  assistance  given  by  the  speakers  for  their  work  in  preparing 
and  delivering  their  presentations  and  in  some  cases  editing  them  for 
publication.  Representatives  of  the  General  Accounting  Office  also  merit 
particular  mention  for  their  assistance  in  planning  the  seminar  as  well 
as  taking  part  in  it. 

Staff  members  of  several  Divisions  of  the  Congressional  Research 
Service  --  in  particular,  Genevieve  J.  Knezo  of  the  Science  Policy  Research 
Division  --  cooperated  in  developing  the  agenda  for  the  meetings.  Gary 
Evans  of  the  CRS  Office  of  Special  Programs  was  the  coordinator  for  the 
seminar  series.  The  proceedings  were  assembled  and  summarized  by  Ms.  Knezo 
with  the  cooperation  of  Mr.  Evans  and  of  members  of  the  Science  Policy 
Research  Division.  The  project  as  a  whole  was  carried  out  under  the 
joint  direction  of  myself  and  Warren  Johnston,  Chief  of  the  CRS  Office 
of  Special  Programs. 

We  have  been  pleased  to  undertake  this  project.  We  hope  the 
proceedings  will  be  of  value  to  you  and  your  staff. 


Sincerely, 


fbrman  Beckman 
Acting  Director 


INTRODUCTION 

A.   Purpose  and  Scope  of  the  Seminar 

On  May  5,  12,  and  19,  1975,  the  Congressional  Research  Service 
sponsored  a  three-day  seminar  series  on  legislative  oversight  and  pro- 
gram evaluation.   The  Service  held  the  series  at  the  request  of  Senator 
Lee  Metcalf,  Chairman  of  the  Senate  Government  Operations  Subcommittee 
on  Reports,  Accounting,  and  Management,  and  Senator  Bill  Brock,  Ranking 
Minority  Member  of  that  Subcommittee.   The  series  had  two  purposes. 
The  first  was  to  acquaint  legislative  staff  who  have  responsibilities 
or  interest  in  legislative  oversight  and  program  evaluation  with 
current  requirements,  processes,  and  resources  for  oversight  and  evalua- 
tion.  The  second  was  to  provide  a  forum  for  discussing  problems  that 
staff  might  encounter  in  conducting  oversight  and  in  doing  or  using 
evaluation  research. 

Speakers  at  the  meetings  consisted  of  committee  staff;  members 
of  the  staff  of  the  General  Accounting  Office  (GAO),  Congressional 
Research  Service  (CRS),  and  Office  of  Technology  Assessment  (OTA); 
officials  from  the  Office  of  Management  and  Budget  and  the  Office  of 
Education;  and  nongovernmental  specialists. 

An  average  of  75  persons  attended  each  of  the  three  all  afternoon 
sessions.   They  were  committee  and  member  staff  aides,  staff  of  congressional 
support  agencies  such  as  the  CRS,  GAO,  and  OTA,  and  staff  of  the  executive 
branch  and  nongovernmental  evaluation  research  communities. 

Shortly  after  the  series  ended,  Senator  Sam  Nunn,  Chairman  of 
the  Subcommittee  on  Oversight  Procedures  of  the  Senate  Committee  on 
Government  Operations  asked  the  CRS  to  prepare  the  proceedings  of 


the  seminar  for  publication.   The  two  objectives  of  the  publication 
would  be:   (1)  to  summarize  the  major  issues  and  preserve  the  pro- 
ceedings for  the  benefit  of  those  legislative  staff  unable  to  at- 
tend the  meetings,  and  (2)  to  compile  a  comprehensive  resource  docu- 
ment on  the  topic  of  oversight  and  evaluation.    1/ 

The  proceedings  consist  of  this  introduction;  a  summary  of  major 
themes  raised  during  the  series;  the  papers  presented  each  day,  pre- 
ceded by  a  summary  of  the  day's  proceedings;  and  appendix  materials 
consisting  of  documents  speakers  used  to  illustrate  their  activities. 

The  papers  are  presented  in  the  order  in  which  they  were  given. 
The  meeting  was  not  recorded;  therefore  highlights  of  discussions  which 
follow  the  papers  are  based  on  notes  taken  during  the  meetings. 

B.   The  Context  of  Evaluation  and  Oversight 

The  current  period  of  increasing  budgetary  restraints  and  com- 
peting claims  on  scarce  Federal  resources  ,  coupled  with  an  increasing 
proliferation  of  Federal  benefit  programs,  has  created  a  climate  conducive 
to  the  adoption  of  better  methods  of  program  planning,  management, 
and  oversight.  During  recent  years  the  agencies  of  the  executive 
branch  have  instituted  several  important  reforms  to  improve  the  ef- 
ficiency and  efficacy  of  Federal  spending.   These  include  both  modi- 
fied and  new  administrative  arrangements,  and  development  of  new 


_l/  Senator  Nunn  indicated  that  this  document  would  be  one  part  of  a 
series  that  the  Subcommittee  on  Oversight  Procedures  expected  to 
publish  on  the  topic  of  improving  resources  and  techniques  of 
oversight.   Another  volume  planned  in  the  series  would  include 
analytical  essays  on  oversight  and  evaluation. 


decisionmaking  and  policy  analysis  techniques.   Technical  reforms  are 
seen  in  the  adoption  of  such  techniques  as:   Management  by  Objectives 
(MBO) ,  program  evaluation,  cost/benefit  analysis,  and  cost/effective- 
ness analysis.   However,  these  new  techniques  are  constrained  by  some 
of  the  same  limitations  which  plagued  their  1960s  precursors,  PPBS 
and  systems  analysis.   These  include:   (1)  unrealistic  expectations 
of  analytical  tools  and  techniques,  whose  imprecision  and  relatively 
unsophisticated  level  of  development  hamper  the  generation  of  wholly 
valid  and  reliable  information;  (2)  inadequacies  of  data  collection 
and  generation  procedures  which  may  vitiate  the  utility  of  tight 
analytical  and  conceptual  schemes;  and  (3)  imprecise  understanding 
of  the  contribution  of  quantitative  analytical  information  to  gov- 
ernmental and  political  decisionmaking. 

During  1972,  Senator  William  V.  Roth  of  Delaware  undertook  a 
study  of  Federal  evaluation  practices.   The  conclusion  of  this  study 
underscored  the  need  for  better  analysis  and  evaluation  of  Federal 
programs,  and,  at  the  same  time,  identified  many  of  the  problems  which 
hamper  effective  analysis  and  evaluation.   One  of  the  paragraphs  of 
the  study  summarizes,  for  both  decisionmakers  and  analysts,  the  lim- 
itations and  ambiguities  of  much  analysis: 

The  use  of  analytical  techniques  is  subject  to  a  number  of  dan- 
gerous distortions.   These  include  over-objectif ication,  over- 
systemization,  and  use  for  advocacy  by  program  managers  and 
political  executives.   We  must  keep  in  mind  that  it  is  especial- 
ly difficult  to  gauge  whether  social  programs  are  successful. 
These  programs  necessarily  have  multiple  goals  which  in  their 
ultimate  form  are  very  hard  to  measure.   Further  I  think  we 


need  to  guard  against  the  erection  of  complicated  formal  struc- 
tures of  analysis  which  have  no  impact  on  decisionmaking.    2/ 

Subsequently,  in  commenting  on  this  study  as  well  as  on  its  own 
commissioned  studies  in  the  area,  the  Joint  Economic  Committee  identi- 
fied another  problem  which  hampers  the  utility  of  quantitative  evalu- 
ation and  analysis:   that  is  the  fact  that  political  decisionmaking 
in  a  democracy  must  include  a  number  of  different  and  sometimes  con- 
flicting types  of  information.   Systematically  generated  analytical 
information  is  only  one  type  of  information  politicians  use  in  making 
decisions: 

Analysis  can  provide  an  important  and  helpful  tool  for  making 
decisions,  but  it  is  no  more  than  a  tool.   Problems  involving 
social  policy  and  value  judgments  must  be  considered  and  weighed 
in  conjunction  with  the  results  of  benefit-cost  analysis,  and 
the  final  decision  made  by  the  human  policymaker.    3/ 

Despite  these  limitations,  attempts  to  improve  the  conduct  and 
utility  of  analysis  and  evaluation  have  continued  unabated  in  recent 
years.   This  is  evidenced  by  several  types  of  events.   First,  aca- 
demics, extramural  policy  researchers  and  analysts,  and  evaluators, 
either  singly  or  in  conjunction  with  governmental  officials,   have 

continued  to  discuss  the  issues  raising  the  level  of  awareness 

and  debate  about  the  conduct  and  utility  of  quantitative  analytical 


2/   Report  on  A  Survey  of  Federal  Program  Evaluation  Practices.   By 
Hon.  William  V.  Roth,  Jr.,  a  U.S.  Senator  from  the  State  of  Dela- 
ware, pp.  1-52  in  U.S.  Congress,  Joint  Economic  Committee,  Bene- 
fit-cost Analysis  of  Federal  Programs.  A  compendium  of  papers 
submitted  to  the  Subcommittee  on  Priorities  and  Economy  in  Gov- 
ernment.  9 2d  Cong.,  2d  sess.   January  2,  1973,  Washington,  U.S. 
Govt.  Print.  Off.,  1973,  p.  quoted  on  p.  53.   (Joint  committee 
print . ) 

3 /  Benefit-Cost  Analysis  of  Federal  Programs,  op.  cit. ,  p.  54. 


techniques,  especially  those  which  are  based  on  the  methods  of  the 
social  sciences.    4/ 

Second,  the  executive  and  legislative  branches'  principal 
fiscal  oversight  agencies,  the  Office  of  Management  and  Budget  and 
the  General  Accounting  Office,  have  considerably  increased  their  ef- 
forts to  improve  the  management,  conduct,  utility,  and  understanding 
of  the  analysis  and  evaluation  research  enterprise.   Both  of  these 
agencies,  as  indicated  in  the  discussions  below,  have  undertaken 
a  number  of  activities  to  develop  policies  to  govern  the  evaluation 
research  process;  to  improve  legislative  language  for  evaluation, 
and  to  forge  better  links  between  evaluation  and  budgeting.   GAO  in 
addition  has  special  responsibilities  for  evaluation  under  the  Congres- 
sional Budget  and  Impoundment  Control  Act  of  1974,  P.L.  93-344. 


_4_/  There  are  numerous  documents  on  this  subject.   Some  of  the  more 
recent  and  general  treatments  are: 

(1)  Beckman,  Norman  and  Harold  Handerson,  eds.  New  Directions  in 
Public  Administration.  Proceedings  of  "Opposing  Forces  in  Public 
Administration,"  The  Fifth  Annual  Conference  of  the  National  Cap- 
ital Area  Chapter  of  the  American  Society  for  Public  Administra- 
tion, November  14-15,  1974.  Reston,  Virginia,  The  Bureaucrat, 
Inc.,  1975:  Part  II.  "Program  Evaluation:  Does  It  Improve  Pub- 
lic Policies  and  Administration?",  pp.  58-98. 

(2)  Lyons,  Gene  M. ,  ed.  Social  Research  and  Public  Policies.  The 
Dartmouth /OECD  Conference.  Dartmouth,  The  Public  Affairs  Center, 
1975.  206  p.  See  especially  Part  II,  Social  Experiments,  pp.  87- 
140  and  "Part  III,  Evaluation  Research,"  pp.  141-205. 

(3)  Knezo,  Genevieve  J.   Program  Evaluation:   Emerging  Issues  of 
Possible  Legislative  Concern  Relating  to  the  Conduct  and  Use  of 
Evaluation  in  the  Congress  and  the  Executive  Branch.   Multilith, 
Science  Policy  Research  Division,  Congressional  Research  Service, 
Library  of  Congress,  Nov.  16,  1974.   79  p.   (75-35  SP.) 


Third,  and  undoubtedly  most  important,  the  Congress  itself  has  adopted 
several  recent  measures  to  improve  the  conduct  of  evaluation  and  oversight 
and  to  better  link  program  planning,  budgeting,  evaluation,  and  oversight. 
These  activities  include:  (1)  passage  of  the  Legislative  Reorganization  Act 
of  1970,  which  strengthened  the  policy  analysis  and  information  capabilities 
of  the  Congressional  Research  Service,  called  upon  the  General  Accounting 
Office  to  make  available  to  all  members  and  committees  of  the  House  and 
Senate  lists  of  GAO  reports  and  when  requested  the  reports  themselves,  and 
required  the  preparation  of  biennial  committee  reviews  of  legislative 
activities;  (2)  passage  of  the  House  Committee  Reform  Amendments  of  1974, 
H.  Res.  988,  which  enlarged  oversight  responsibilities  of  standing  committees, 
especially  of  the  House  Government  Operations  Committee;  and  (3)  passage  of 
the  Congressional  Budget  and  Impoundment  Control  Act  of  1974,  P.L.  93-344, 
which  authorized  committees  to  conduct  or  make  use  of  evaluation  and 
augmented  GAO's  capability  to  aid  committees  in  conducting  evaluation 
and  in  writing  requirements  and  legislative  language  for  evaluation. 
Other  related  activities  include:  creation  of  the  House  Information  Systems 
Group,  which  provides  members  and  staff  with  access  to  models  and  data 
useful  for  evaluation  purposes;  creation  of  the  Office  of  Technology 
Assessment,  to  provide  the  Congress  with  unbiased  information  and  analysis 
regarding  the  positive  and  negative  impacts  of  technological  applications; 
creation  of  functionally  distinct  oversight  committees  in  the  Senate 
Committee  on  Government  Operations;  and  a  study  of  the  need  to  revamp 
the  Senate  committee  structure.   Informal  activities  along  these  lines, 
including  several  sponsored  by  Senator  Bill  Brock,  consist  of  a  monthly 
seminar  for  legislative  staff  on  program  evaluation,  and  preparation  of 
a  draft  manual  on  program  evaluation  concepts  and  techniques  to  assist 


legislative  staff  who  have  responsibilities  for  writing  requirements  for 
evaluation  into  new  legislation. 

As  recently  as  five  years  ago,  evaluation  units  in  the  Federal  Gov- 
ernment were  rare.   Today  they  are  widespread,  __5_/  and  by  the  latest 
OMB  account,  Federal  agencies,  exclusive  of  the  Departments  of  De- 
fense and  Treasury,  are  responsible  for  spending  almost  $120  million 
annually  for  evaluation  research  studies.    6/   Evaluation  research  is 
a  relatively  new  management  and  policy  research  tool,  widely  used,  but 
as  evidenced  in  the  CRS  seminar  and  in  other  forums,  beset  with  the 
problems  which  accompany  any  new  decisionmaking  technique.   These 
problems  cannot  be  solved  without  considerable  effort  by  both 
evaluation  performers  and  users.   In  concluding  the  November  1974  Amer- 
ican Society  for  Public  Administration  seminar  on  evaluation  research, 
Keith  E.  Marvin  of  the  General  Accounting  Office  noted  in  this  respect: 
".  .  .Despite  the  continued  existence  of  some  basic  problems  and  ques- 
tions concerning  evaluation  work,  there  will  be  more  such  activity  at 
both  the  agency  and  the  legislative  staff  level  in  the  future.  What  is 
accomplished  will  become  at  least  as  important  as  how  well."  ]_/ 


5/  Lafayett,  David  P.,  et.  al.  "Federal  Department  and  Agency  Ex- 
periences Using  Program  Evaluation."  In  Beckman,  New  Directions 
in  Public  Administration:   The  Federal  View,  op_.  cit. ,  p.  69. 

_6_/  U.S.  Office  of  Management  and  Budget.  Evaluation  and  Program 
Implementation  Division.  Survey  of  Federal  Evaluation  Activi- 
ties, Sept.  1975.   Washington,  U.S.  Govt.  Print.  Off.,  1975.  246  p. 

Jj      "Program  Evaluation:   Does  It  Improve  Public  Policies  and  Admin- 
istration:  Summary  of  Panel  and  Floor  Discussions."  In  Beckman, 
New  Directions  in  Public  Administration:   The  Federal  View,  op_. 
cit.,  p.  97. 


The  CRS  seminar  on  Legislative  Oversight  and  Program  Evaluation 
was  designed  to  provide  a  forum  for  identifying  and  discussing  some 
of*  the  major  issues  surrounding  evaluation  and  oversight,  especially 
in  relation  to  the  responsibilities  of  legislative  staff.   The  seminar 
addressed  topics  of  both  production  and  implementation,  such  as  method- 
ology, data,  funding  level,  procurement,  type  of  contractor,  training, 
timing  of  evaluation,  intelligibility  of  reports,  and  use  of  qualita- 
tive and  quantitative  evaluation  information  in  legislative  decision- 
making and  oversight.   Neither  the  summary  which  follows  nor  the  semi- 
nar were  designed  to  resolve  issues  regarding  evaluation  and  oversight, 
but  to  raise  the  level  of  awareness  and  debate  on  the  issues  as  a  modest 
step  toward  improving  the  efficiency  and  efficacy  of  Government. 


SUMMARY  OF  THE  MAIN  THEMES  OF  THE  SEMINAR  SERIES  ON  "LEGISLATIVE 
OVERSIGHT  AND  PROGRAM  EVALUATION,"  MAY  1975 


A.   The  Scope  and  Agenda  of  the  CRS  Seminar  on  Legislative  Oversight  and 
Program  Evaluation 

Each  seminar  session  was  organized  around  a  separate  theme.   The 
first  afternoon's  presentations  overviewed  general  responsibilities 
and  resources  for  oversight  and  evaluation.   House  and  Senate  Govern- 
ment Operations  Committee  staff,  as  well  as  staff  from  GAO,  CRS,  and 
OTA,  made  presentations  and  assisted  in  answering  questions. 

Congressional  committee  experiences  in  conducting  evaluations  or 
using  secondary  policy  analyses  or  evaluations  done  by  congressional 
support  agencies,  or  quantitative  analytical  evaluation  research  re- 
ports performed  by  or  for  executive  agencies ,  formed  the  theme  of  the 
second  seminar  session.   The  substantive  issues  addressed  were  over- 
sight and  evaluation  of  foreign  policy,  by  the  House  Committee  on 
International  Relations;  implementation  of  the  Freedom  of  Information 
Act  by  the  Subcommittee  on  Foreign  Operations  and  Government  Infor- 
mation of  the  House  Committee  on  Government  Operations;  examination 
of  the  Joint  Economic  Committee's  extensive  investigations  and 
studies  of  public  welfare;  and  committee  evaluation  and  oversight  by 
the  Senate  Committee  on  Agriculture  and  Forestry  of  the  Farmers  Home 
Administration  and  other  agricultural  programs. 

The  third  seminar  session  had  a  double  theme:  (1)  the  state-of-the- 
art  of  evaluation  and  (2)  executive  branch  evaluation  activities. 
Speakers  addressed  the  pros  and  cons  of  various  evaluation  research 
methods;  efforts  to  enhance  oversight  of  federally  funded  evaluation 
research;  and  efforts  of  the  GAO  and  Office  of  Management  and  Budget 
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to  refine  executive  agency  evaluation  units  and  processes.   Two  members 
of  Senator  Brock's  staff  described  their  monthly  evaluation  seminar  for 
legislative  staff  and  their  manual  on  evaluation,  for  legislative  users. 
The  agenda  also  included  a  case  study  of  evaluation  in  the  Office  of 
Education,  and  a  presentation  by  an  expert  on  evaluation  research  methodology. 

B.  Introduction  to  Oversight 

Walter  Oleszek  of  the  Congressional  Research  Service  defined  over- 
sight as  the  exceedingly  complex  process  of  determining  whether  executive 
agencies  carry  out  programs  as  the  Congress  intended.   He  said  that  in 
the  past,  the  Congress  had  experienced  difficulties  in  conducting  effective 
oversight.   These  difficulties  were  attributable  to  several  factors,  in- 
cluding: 

-  a  lack  of  staff  with  time  for  oversight,  since  most  staff  efforts 
had  to  be  used  to  write  new  legislation; 

-  a  tendency  to  avoid  formal  in-depth  oversight  because  of  the 
advocacy  relationships  which  might  have  developed  between  some  committees 
and  the  agencies  they  oversee;  and 

-  a  lack  of  established  procedures  for  cooperation  between  Government 
Operations  Committees  and  authorizing  committees  in  sharing  oversight 
information.   8/  Other  speakers  also  identified  potential  obstacles  to 
effective  evaluation.   These  included:  a  reluctance  on  the  part  of  some 
Federal  agencies  to  doing  evaluation  research  whose  results  might  ultimately 


_8_/  Oleszek,  Walter.   "Congressional  Oversight,  A  Review  of  Recent 
Legislative  Actions,"  Day  1. 
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affect  agency  budgets  and  staff  size;  _9_/  and  difficulties  both  in 
determining  which  evaluation  techniques  most  appropriately  answer  over- 
sight questions  and  in  pinpointing  policy  options  that  evaluation  and 
oversight  studies  might  generate.   10/ 

C.  Types  of  Oversight 

There  was  an  implied  consensus  about  several  different  types  of 
oversight  and  techniques  which  can  be  used  to  support  these  activities. 
Walter  Oleszek  differentiated  among  three  types  of  oversight.   "Legisla- 
tive oversight"  is  done  by  authorizing  committees  to  determine  whether 
agencies  have  properly  implemented  programs  under  their  jurisdiction. 
"Fiscal  oversight,"  primarily  a  function  of  the  Appropriations  Committees, 
scrutinizes  Federal  spending.   "Investigative  oversight,"  until  recently 
a  responsibility  primarily  of  the  House  and  Senate  Government  Operations 
Committees,  examines  programs  which  cut  across  agency  and  congressional 
committee  jurisdictions.   A  committee  staff  member  added  definitions  for 
other  more  specific  and  "operational"  types  of  congressional  oversight. 
"Routine  oversight"  consists  of  daily  monitoring  of  Federal  agency  regula- 
tions to  determine  whether  the  agency  is  following  the  intent  of  the  law. 


9/  See  especially  papers  and  discussions  of:  Giltmier,  James  W.  "Policy 
Formation  Through  Program  Evaluation  and  Systems  Analysis:  A  Congres- 
sional View,"  Day  2;  and  Schick,  Allen,  "Evaluating  Evaluation:  A 
Congressional  Perspective,"  Day  3. 

10/  See  especially  Knezo,  Genevieve  J.,  "Program  Evaluation  Concepts." 
On  policy  options  related  to  evaluation,  especially  the  need  for 
"decision-driven  evaluation,"  see  Graves,  Clifford  W. ,  remarks  based 
on  Federal  Program  Evaluation:  The  Perspective  From  0MB,"  Day  3; 
and  Abt,  Clark  C.   "The  State  of  the  Art  of  Program  Evaluation," 
Day  3." 


12 


"Special  oversight,"  like  investigative  oversight,  involves  special 
in-depth  investigation  of  a  high  priority  area  which  might  result  in 
the  need  for  corrective  legislative  action.   "Guerilla  oversight"  consists 
of  spot-checking,  in  a  one  day  hearing,  where  adversary  proceedings  can 
lead  to  corrective  action  by  an  agency,  but  without  legislative  enactment 
or  amendment.   11/ 

D.  Types  of  Evaluation  Research 

Federal  agency  administrators  generally  conduct  or  contract  for  one 
of  two  types  of  quantitative  evaluation  studies:  process  evaluations  or 
impact  evaluations.   These  are  done  at  the  request  of  the  Congress  or 
for  the  agency's  own  management  purposes.   These  types  of  evaluation  are 
used  for  different  purposes  and  are  grounded  in  different  types  of  techniques, 
Process  evaluations  are  used  to  answer  management  questions.   Generally 
process  evaluations  use  non-experimental  evaluation  techniques  and  data 
generated  in  the  normal  course  of  administering  and  monitoring  a  program. 
These  types  of  evaluation  are  far  less  costly  and  require  far  less  time 
to  do  than  do  impact  evaluations.   Impact  evaluations  answer  questions 
about  whether  a  program  caused  a  change  in  attitude  or  behavior  in  the 
participants  or  targets  of  a  program.   Impact  evaluations  are  more 
difficult  to  do  than  process  evaluations.   Generally  they  require 
experimental  research  techniques  and  randomized  test  and  control  groups. 


11/  Remarks  of  Phillips,  William  G.   "The  Freedom  of  Information  Act, 
Day  2. 
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New  and  costly  data  collection  procedures  must  be  built  into  a  program, 
usually  in  the  program  authorization  stage,  to  meet  the  requirements  of 
impact  evaluation.   12/ 

E.  Scope  and  Limitations  of  Evaluation  Research  for  Legislative  Oversight 

It  was  noted  that  impact  and  process  evaluation  are  useful  for  many 
types  of  legislative  oversight  —  but  most  especially  for  those  types 
dealing  with  program  formulation  and  amendment.  13/  However,  most 
speakers  seemed  to  agree  that  information  generated  using  any  one  specific 
evaluation  technique  constitutes  only  one  type  of  information  that  members 
use  in  making  political  decisions.   On  this  point,  Clark  Abt,  emphasized: 
"What  we  cannot  do  with  program  evaluation  is  create  new  programs,  make 
decisions,  make  the  assumption  of  rationality  come  true,  and  get  users 
or  consumers  of  program  evaluations  to  ask  the  right  questions.   We  also 
cannot  make  value  judgments  that  have  any  credibility  where  these  are 
decisive  program  decisions."   14/ 

However,  the  point  was  made  repeatedly  that  many  evaluation  research 
studies  are  done  improperly,  as  a  result  they  fail  to  yield  information 
useful  for  decisionmaking.   Several  factors  contribute  to  this  situation. 


12/  See  Knezo,  op.  cit.  Abt,  op.  cit.,  elaborated  upon  these  themes. 

13/  Foskett,  William  H.   "Practical  Application  of  Evaluation  in  Leg- 
islative Processes,"  Day  3. 

14/  Abt,  op.  cit.,  Day  3. 
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First,  the  need  to  represent  different  political  viewpoints  usually 
produces  broad  or  ambiguous  statements  of  legislative  program  goals.  As 
a  result  evaluators  have  only  imprecise  yardsticks  against  which  to 
measure  program  or  agency  accomplishments.  15/ 

Second,  evaluation  researchers  and  program  managers  often  lack 
awareness  of  the  questions  that  evaluation  and  oversight  can  and  cannot 
answer  and  the  proper  techniques  to  use  to  answer  these  questions. 
Legislative  staff  are  not  always  aware  of  these  issues.  16  / 

In  relation  to  this  issue,  Allen  Schick  suggested  that  the  Congress 
should  be  wary  of  getting  directly  into  the  business  of  carrying  out 
program  evaluations.   His  rationale  was  that  legislative  oversight  is  not 
analogous  to  program  oversight,  which  requires  specialized  evaluation 
techniques  and  should  be  a  responsibility  primarily  of  the  Federal  agencies 
which  administer  programs.  17/ 

Several  legislative  staff  speakers,  however,  emphasized  the  need 
for  legislative  program  evaluation.   A  few  gave  illustrations  of  committees 
conducting,  monitoring,  or  using  evaluations.   They  pointed  out  that  Title 
Seven  of  the  Congressional  Budget  and  Impoundment  Control  Act,  P.L.  93-344, 
authorizes  committees  to  conduct,  contract  for,  or  request  executive 
agencies  to  do  evaluations.   It  was  also  noted  that  executive  agencies 
are  increasingly  calling  for  the  conduct  of  evaluation  research.   A  recent 


15/  See  especially  remarks  of  Little,  Dennis  L.   "Ask  the  Right  Questions,' 
Day  1,  and  Abt,  and  Oleszek,  op.  cit. 

16/  See  presentations  of  Foskett,  op.  cit.,  and  Abt,  op.  cit.,  especially. 

17/  Remarks  of  Schick,  op.  cit.,  Day  3. 
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General  Accounting  Office  study,  for  instance,  demonstrated  the  federally 
funded  program  evaluation  research  studies  cost  at  least  $130  million  in 
the  Fiscal  Year  1974.   It  was  also  noted  that  because  of  the  Budgeting  Act 
and  increased  oversight  responsibilities  in  the  House,  committees  are 
increasingly  writing  legislative  requirements  for  program  evaluation  into 
bills  authorizing  new  programs  —  requiring  staff  to  be  intimately  familiar 
with  the  promises  and  perils,  methods,  and  utility  of  different  types  of 
program  evaluation  research  studies. 

There  was  general  agreement  that  the  Congress  has  limited  resources, 
cannot  and  does  not  desire  to  compete  with  or  duplicate  the  information 
gathering  and  analysis  capabilities  of  the  executive  branch,  and  has 
oversight  information  requirements  different  from  the  information  require- 
ments of  Federal  executive  agencies  which  administer  programs.   Therefore, 
committee  staff  with  responsibilities  for  oversight  and  evaluation  might 
be  more  selective  in: 

-  determining  priorities  for  oversight  and  evaluation,  and 

-  using  their  own  staff  or  the  staff  of  the  auxiliary  legislative 
information  and  analysis  support  bodies,  as  well  as  the  staff  of  Federal 
agencies,  to  review  existing  literature  to  determine  if  there  already  is 
information  available  to  assist  in  developing  priorities  for  evaluation 
and  oversight. 

F.  Illustrations  of  Committee  Oversight  and  Evaluation  Activities 

The  case  studies  of  oversight  and  evaluation  presented  during  the 
seminar  illustrated  a  variety  of  qualitative  and  quantitative  information 
gathering  and  analysis  techniques  available  to  committee  and  requirements 
for  their  successful  use. 
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Dr.  Alair  Townsend,  a  former  staff  member  of  the  Subcommittee  on 
Fiscal  Policy  of  the  Joint  Economic  Committee  discussed  a  particularly 
comprehensive  oversight  activity.   In  its  role  as  an  investigative 
committee  serving  both  the  House  and  the  Senate,  the  Joint  Economic 
Committee  conducted  a  series  of  studies  and  hearings  to  assess  both 
process  (management)  and  impact  (program  effectiveness)  issues  relating 
to  the  Family  Assistance  Plan.   To  do  this  adequately,  however,  the  Com- 
mittee had  to  assess  the  Family  Assistance  Plan  in  the  context  of  all 
income  assistance  programs.   A  series  of  committee  prints  dealing  with 
these  topics  was  prepared  by  in-house  staff,  outside  contractors,  staff 
of  Federal  agencies,  and'  legislative  support  agencies,  such  as  the 
Congressional  Research  Service  and  the  General  Accounting  Office. 

Several  important  observations  were  made  about  requirements  for 
the  conduct  of  studies  of  this  nature.   First  among  these  was  the  need 
for  a  highly  trained  committee  staff  to  monitor  closely  the  work  of 
congressional  support  agencies  and  outside  contractors.   This  is 
necessary  to  keep  the  work  on  schedule,  and,  perhaps  more  important  to 
insure  that  it  conforms  to  a  legislative  decisionmaker's  needs.   This 
type  of  monitoring  is  indispensable  in  the  case  of  certain  outside 
contractors,  for  example  those  social  scientists  who  tend  to  use  Federal 
research  monies  more  to  train  students  and  to  produce  basic  or  applied 
research  valued  by  their  peers  than  to  produce  research  useful  to  policy- 
makers.  Close  monitoring  of  outside  contractors  is  required  also  to  insure 
that  reports  are  not  too  laden  with  academic  jargon.   18/ 


18/  See  remarks  and  discussion  following  the  presentation  of  Townsend, 
Alair  A.  "Congressional  Committee  Case  Studies:  Studies  in  Pub- 
lic Welfare,"  Day  2. 
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John  H.  Sullivan  Illustrated  another  unique  legislative  oversight 
device  in  his  presentation  describing  some  of  the  oversight  activities 
of  the  House  Committee  on  International  Relations  and  the  Senate  Committee 
on  Foreign  Relations.   These  committees  use  overseas  field  staff  visits 
by  highly  trained  committee  staff  investigators.   One  of  the  principal 
advantages  of  this  method  of  information  gathering  is  that  it  generates 
information  that  can  rarely  be  obtained  any  other  way.   It  is  also  one 
of  the  most  direct  means  of  supplying  oversight  information  because 
research  is  done  by  staff  members  who  have  the  trust,  confidence,  contact, 
and  rapport  with  members  required  to  insure  ready  access  to  policymaking. 
However,  effective  overseas  field  visits  also  require  that  staff  be  highly 
trained  and  conduct  visits  in  pairs  to  corroborate  verbal  evidence  which 
executive  officials  might  deny  later  if  called  to  testify.   19/ 

Various  other  oversight  and  evaluation  techniques  were  described  in 
the  case  studies  presented  during  the  second  day  of  the  seminar.   These 
included  the  conduct  of  program  evaluation  research  by  staff  of  the 
Senate  Committee  on  Agriculture  and  Forestry,  the  use  of  public  opinions 
polls  by  the  Senate  Committee  on  Government  Operations  to  obtain  informa- 
tion for  investigative  oversight  in  the  area  of  intergovernmental  relations, 
and  the  relevance  to  congressional  oversight  of  the  subpoena  power,  as 
described  by  staff  of  the  Senate  Committee  on  Government  Operations. 


19/  See  remarks  and  discussions  of  presentation  by  Sullivan,  John  H. 
"Foreign  Affairs  Oversight,"  Day  2. 
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G.  Use  of  Congressional  Support  Agencies 

Several  illustrations  were  given  of  instances  where  committees  used 
a  variety  of  support  services  provided  by  the  Congressional  Research 
Service,  the  General  Accounting  Office  and  the  Office  of  Technology 
Assessment  to  assist  in  oversight  and  evaluation.   They  included,  for 
example,  assessment  by  the  CRS  of  the  quality  and  extent  of  use  of 
executive  branch  reports  to  Congress  as  required  in  foreign  affairs 
legislation;  the  use  of  the  CRS  to  analyze  agency  responses  to  a  committee 
questionnaire  on  implementation  of  the  Freedom  of  Information  Act;  and 
the  use  of  the  Service  in  preparing  analytical  reports  on  the  Family 
Assistance  Program.   Because  of  its  long  history  in  doing  auditing  and 
oversight  work  for  the  Congress,  the  General  Accounting  Office  was  fre- 
quently cited  as  a  useful  information  and  analysis  source.   Instances 
of  GAO  assistance  ranged  from  illustrations  of  its  aid  in  overseas 
investigations  to  assistance  in  evaluating  the  Farmers  Home  Administration. 

It  was  indicated,  however,  that  there  are  constraints  on  the  use  of 
these  legislative  support  resources  similar  to  the  limitations  on  the  use 
of  other  outside  evaluation  resources.   These  constraints  consist  primarily 
of  the  need  for  close  committee  staff  monitoring  to  insure  that  research 
is  done  in  terms  and  in  time-frame  relevant  to  decisionmakers.   20/ 

1.   Office  of  Technology  Assessment.   Dr.  Gary  Thomas  of  the  Office 
of  Technology  Assessment  noted  that  the  OTA  has  adopted  a  policy  of 
interfacing  closely  with  committee  staff,  not  only  to  receive  suggestions 
for  assessment  topics  but  also  to  help  design  research  appropriate  to 


20/  See  papers  of  end  discussions  following  the  papers  of  Giltmier, 
op.  cit. ,  and  Townsend,  op.  cit. 
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legislative  needs.   The  OTA  ensures  this  interface  by  providing  requesting 
committees  with  periodic,  sometimes  monthly,  progress  reports  from  con- 
tractors during  the  course  of  a  technology  assessment  to  assist  if  necessary 
in  meeting  any  immediate  legislative  decisionmaking  needs.   Also  in  support 
of  this  objective  the  OTA  has  begun  to  do  mini-assessments  or  exploratory 
assessments  in  addition  to  the  longer-range  studies  it  usually  funds. 
Mini-assessments  have  either  one  of  two  purposes:  (1)  to  determine  whether 
a  more  in-depth  assessment  may  indeed  be  required;  or  (2)  to  do  a  short 
two-week  inhouse  study  to  meet  members'  immediate  information  needs.   Such 
a  study  was  done  recently  to, assist  members  in  reviewing  the  budget  of  the 
Energy  Research  and  Development  Agency,  ERDA.  21/ 

2.   Congressional  Research  Service.   The  basic  authority  for  the 
Congressional  Research  Service  to  support  legislative  oversight  and 
program  evaluation  comes  from  the  Legislative  Reorganization  Act  of  1970. 
The  CRS  provides  several  different  types  of  support  to  fulfill  its  respon- 
sibilities. Among  those  described  at  the  seminar  were:  policy  analysis 
reports,  reports  dealing  with  terminating  issues  and  emerging  issues; 
seminars  on  important  public  policy  issues;  the  preparation  of  issue  briefs; 
formation  of  a  futures  group;  and  refinement  of  automated  information 
systems  capabilities  including  bibliographic  citations  by  subject,  bill 


21/  See  presentation  of,  and  discussion  following,   Thomas,  Gary,  "The 
Office  of  Technology  Assessment:  A  Description  of  the  Office's 
Objectives  and  Activities,"  Day  1. 
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digests,  access  to  technical  information  and  bibliographic  data  bases 
including  MEDLARS,  MEDLINE,  and  legal  codes  and  econometric  models.  22/ 

3.   The  General  Accounting  Office.   The  General  Accounting  Office 
recently  instituted  several  activities  intended  to  improve  its  oversight 
and  evaluation  support  capabilities.   These  include,  for  example,  plans 
to  establish  a  Congressional  Information  Services  Group  to  coordinate 
and  help  standardize  social  program  information  required  for  evaluation 
and  to  improve  Congress1  and  GAO's  capabilities  to  do  and  use  program 
evaluations  as  required  under  the  provisions  of  P.L.  93-344.   In  addition, 
the  agency  is  preparing  an  inventory  of  evaluation  studies  done  for 
Federal  agencies  as  well  as  developing  methodological  criteria  and  standards, 
and  legislative  language  formats  for  committees  to  assist  them  in  making 
the  tradeoffs  required  between  different  types  of  evaluation  strategies 
in  relation  to  specific  evaluation  decisionmaking  needs.  23/ 

Some  discussion  was  given  to  potential  limitations  on  the  use  of 
the  General  Accounting  Office  in  assisting  with  legislative  evaluation 
and  oversight.   First,  it  was  noted  that  Federal  agency  administrators 
tend  to  be  less  cooperative  and  informative  with  GAO  staff  than  with 
committee  staff.   This  occurs,  one  staff  person  said,,  because  Federal 


22/  See  the  following  papers:   Beckman,  Norman,   "Congressional 
Research  Service:  Resources  for  Oversight  and  Evaluation," 
Day  1;   Gushee,  David  E.   "The  Service's  Responsibilities  for 
Emerging  Issues  and  Terminating  Programs,"  Day  1;  and  Collier, 
Ellen  C.   "Monitoring  Reports  to  Congress  Required  by  Law,"  Day  1. 

23/   See  presentations  and  discussions  following:  Oliver,  James  P. 
"Remarks  on  the  General  Accounting  Office,"  Day  1;  and  Marvin, 
Keith  E.,   "Current  Evaluation  Activities  in  the  General  Accounting 
Office,"  Day  3.   GAO's  report  on  methodological  standards  was 
published  as:  U.S.  General  Accounting  Office.   Exposure  Draft: 
Evaluation  and  Analysis  to  Support  Decisionmaking.   December  9,  1975 
52  p.   (OPA076-9). 
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administrators  attribute  less  power  to  GAO  staff  than  to  committee  staff. 
It  was  also  noted  that  sometimes  there  is  a  long  "turnaround"  time  for 
the  completion  of  some  GAO  reports,  which  may  take  a  year  to  complete 
and  often  require  a  time-consuming  seven-step  review  process.   In  response, 
agency  and  legislative  staff  noted,  the  GAO  has  begun  to  institute  informal 
reporting  procedures  which  reduce  the  review  time  and  provide  staff  with 
copies  of  draft  reports.   However,  it  was  also  noted  that  staff  sometimes 
use  these  reports  as  if  they  were  the  final  version  of  a  study,  thereby 
jeopardizing  the  integrity  of  the  oversight  process.   24/   It  was  also 
pointed  out  that  some  GAO  reports  customarily  result  from  statutory  re- 
quirements enacted  in  the  past.   As  a  result  the  GAO  might  not  always 
have  resources  available  to  prepare  reports  on  new  issues  currently  before 
the  Congress.  25/ 

H.   Evaluation  in  Executive  Agencies 

Several  of  the  speakers  described  problems  in  the  conduct  and  use 
of  Federally  funded  evaluation  studies.   For  instance,  one  committee 
staff  person  noted  that  Federal  agencies  frequently  refuse  to  release 
evaluation  reports  to  the  Congress.   It  was  also  remarked  that  some 
agency  administrators  tend  to  avoid  objective  evaluation  because  they 
are  the  captive  of  their  programs  and  would  like  to  believe  that  pro- 
grams are  indeed  benefiting  clients.   Several  speakers  noted  that  often 
in  the  past  neither  administrators  nor  some  members  of  Congress  wanted 
evidence  to  demonstrate  that  a  program  might  be  ineffective.   The 


24/  Discussions  and  papers,  Day  2,  especially  Giltmier,  op.  cit. 
25/  Idem. 


22 


gravity  of  this  situation  is  compounded  by  the  fact  that  most  evaluations 
are  done  on  new,  innovative  social  programs,  not  on  long  established  pro- 
grams which  may  profit  more  from  good  evaluation,  but  which  may  be  more 
costly  in  terms  of  possible  bureaucratic  retrenchment.  26/ 

Some  discussion  focused  on  the  issue  of  properly  institutionalizing 
the  locus  of  performance  of  evaluation  within  an  agency  to  enhance  its 
use.   One  speaker  suggested  that  agencies  should  separate  evaluation  from 
budgeting,  because  agencies  usually  try  to  maximize  their  budgets  and 
retain  programs.   Therefore,  the  possibly  useful  adversarial  nature  of 
some  evaluation  is  submerged  to  other  interests.   This  speaker  also 
suggested  that  evaluation  might  be  performed  best  in  a  separate  inspector 
general  office  within  an  agency  due  to  its  critical  adversary  nature.  27  / 
However,  others  submitted  variants  on  these  recommendations.   For  instance, 
one  staff  person  suggested  that  evaluation  would  be  used  more  effectively 
if  it  were  sponsored  by  an  agency  attached  to  the  Office  of  the  Secretary 
of  a  Department.   John  Evans  described  a  variant  used  in  the  Office  of 
Education:  the  project  manager  for  an  evaluation  reads  the  study,  analyzes 
its  policy  implications,  and  sends  a  memo  to  OMB,  via  the  Commissioner 
of  Education,  identifying  the  policy  implications.  28/  Several  speakers 
agreed  in  principle  with  this  comment,  noting  that  agencies  and  legislative 


26/   See  presentations  and  discussions  if  any,  following  the  presentations 
of  Giltmier,  op.  cit.,  Day  1,  Phillips,  op.  cit.,  Day  2,  Oleszek, 
op.  cit.,  Day  1,  and  Schick,  op.  cit.,  Day  3. 

27/   Schick,  op.  cit.,  Day  3. 

2fl/   See  discussion  following  and  paper  of  Evans,  John,  Remarks  based  on 
"Evaluating  Education  Programs  —  Are  We  Getting  Anywhere?"  Day  3. 


23 


staff  should  make  a  special  effort  to  assure  the  conduct  of  "decision- 
driven"  evaluations,  which  are  designed  specifically  to  generate  infor- 
mation for  policymaking.  29/  All  generally  agreed  that  agencies  should 
strive  to  improve  the  development  of  more  useful  evaluation  report  dis- 
semination mechanisms,  both  within  agencies  and  to  the  public. 

With  respect  to  the  development  of  better  policies  to  govern 
evaluation  research,  it  was  noted  that  the  GAO  and  the  OMB  are  doing 
a  joint  study  to  collect  information  about  evaluation  research  in  the 
executive  branch,  including  an  inventory  of  funding  for  evaluation, 
purposes  of  the  evaluation  report,  and  identification  of  the  offices 
supporting  such  work.   The  OMB  is  also  undertaking  several  activities 
to  improve  the  conduct  of  evaluations  within  agencies.   These  include 
an  assessment  of  the  uses  of  agency  evaluations,  an  assessment  of 
evaluation  training  needs,  and  publication  of  a  technical  manual  on 
evaluation  management  systems.  30/ 

I.  Methodological  Needs 

Clark  Abt  raised  several  points  about  how  the  Congress  and  agencies 
might  improve  the  methodological  sophistication  of  evaluation  research. 
In  summary,  these  were:  more  training  for  evaluation  researchers;  Federal 
sponsorship  of  more  cross-cutting  evaluations,  which  attempt  to  assess 
program  impacts  in  a  variety  of  geographic  locations  and  demographic 


29/  See  especially  statements  of  Graves,  op.  eit.,  Day  3  and  Abt, 
op.  cit. ,  Day  3 . 

30/  An  illustration  of  one  OMB  report  of  this  nature  is:  Survey  of 
Federal  Evaluation  Activities.   By  Evaluation  and  Program 
Implementation  Division,  Office  of  Management  and  Budget.   September 
1975.  246  p. 
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groups;  better  Federal  agency  preparation  of  requests  for  proposals 
(RFPs) ;  and  improved  Federal  specification  of  program  goals,  policy 
options,  or  information  needs  that  evaluation  can  support.  31/ 

J.   Evaluation  Requirements 

Clifford  Graves,  Deputy  Associate  Director,  Evaluation  and  Program 
Implementation  Division  of  the  Office  of  Management  and  Budget,  summed 
up  in  his  oral  remarks  the  difficulties  the  Congress  faces  in  using  or 
contracting  for  evaluations.   He  noted  that  an  OMB  assessment  of  Federal 
agency  evaluations  indicated  that  evaluations  generally  are  untimely, 
inaccurate,  and  irrelevant  for  decisionmaking.  32/ 

In  this  respect,  several  speakers,  including  executive  officials 
and  contractors  as  well  as  committee  staff  people  who  have  used  evalu- 
ations or  who  have  been  instrumental  in  heightening  legislative  staff 
awareness  of  the  scope  and  limitations  of  program  evaluation,  outlined 
points  that  congressional  staff  might  consider  when  doing  or  using 
evaluations  or  writing  legislative  language  requiring  evaluation  for 
oversight  purposes.   These  points  may  be  summarized  as  follows.   Staff 
might  consider  the  needs  to: 

-  Set  priorities  for  doing  evaluations.   In  addition,  staff  or 
contractors  should  be  encouraged  to  review  existing  evaluation  literature 
to  see  if  studies  have  already  been  done  on  the  same  or  related  programs. 


31/  Abt,  op.  cit.,  Day  3. 
32/  Graves,  op.  cit.,  Day  3. 
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Doing  so  is  necessary  to  help  formulate  questions,  to  review  the  state 
of  the  art  of  evaluation  methods,  and  to  determine  what  kind  of  evalua- 
tion is  needed.   It  also  helps  to  encourage  fiscal  responsibility.   Staff 
might  also  seek  evaluations  of  past  program  enactments,  which  consume  con- 
siderable Federal  expenditures,  as  opposed    only  to  new  program  enact- 
ments. 

-  Develop  appropriate  statements  of  agency  or  program  goals  to 
assist  in  developing  language  for  later  evaluation  efforts.   This  may 
be  a  potentially  troublesome  problem,  since  legislators  frequently  use 
vague  or  ambiguous  statements  of  program  objectives  for  the  sake  of 
political  compromise.   Sometimes  vague  statements  of  program  goals 
hamper  effective  evaluation  because  evaluators  are  left  with  imprecise 
yardsticks  against  which  to  compare  actual  program  effectiveness  and 
accomplishment . 

-  Determine  whether  a  process  or  an  impact  evaluation  is  required. 
This  is  an  important  consideration  that  must  account  for  the  following 
factors:  what  are  the  questions  which  decisionmakers  want  answered 
about  the  program?  Are  they  mainly  questions  regarding  program  manage- 
ment, or  are  they  questions  regarding  the  actual  effects  or  impacts  of 
the  program  on  its  intended  objects?  When  do  the  decisionmakers  need 
these  answers?  Are  decisionmakers  willing  to  endure  the  possible 
political  side-effects  of  impact  evaluations  or  experimental  evaluations 
required  to  answer  cause  and  effect  questions?   Tradeoffs  must  be  made 
among  all  these  factors. 

-  Be  aware  of  the  possible  policy  options  available  to  legislators 
which  might  result  from  evaluations,  and  write  legislative  language  for 
evaluation  which  is  designed  specifically  to  support  the  generation  of 
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information  and  analysis  relevant  to  those  policy  options  which  are 
available.   It  was  noted  in  this  respect  that  the  General  Accounting 
Office  is  now  surveying  Federal  statutes  which  require  evaluation  to 
develop  standards  adequate  for  generating  appropriate  legislative 
language  for  evaluation. 

-  Determine  the  amount  of  lead-time  available  for  evaluation  in 
relation  to  possible  decisionmaking  options.   It  should  be  remembered 
that  more  time  is  required  for  the  conduct  of  complex,  longitudinal, 
quantitative  evaluations  of  ongoing  programs  than  for  qualitative  or 
non-longitudinal  evaluations. 

-  Recognize  that  impact  evaluations  usually  cannot  employ  data 
collected  in  the  normal  course  of  monitoring  a  program.   If  impact 
evaluations  are  required,  the  legislative  language  for  evaluation  might 
include  the  requirement  of  developing  appropriate  new  data  systems,  as 
well  as  authorization  for  funding  of  these  systems.   However,  staff 
should  be  aware  of  the  possible  disadvantages  of  such  data  collection 
procedures,  including  invasion  of  privacy  and  the  ethical  Issues  of 
experimentation  with  human  beings. 

-  Monitor  very  closely  the  conduct  of  evaluation,  whether  by 
inhouse  staff,  a  congressional  support  agency,  or  an  executive  branch 
agency,  to  insure  that  evaluation  reports  generate  information  and 
analysis  to  answer  the  questions  that  legislative  decisionmakers  want 
answered  about  the  program. 

-  Develop  appropriate  ways  to  disseminate  information  to  members 
and  require  that  summary  evaluations  reports  for  members  be  readable, 
short,  and  free  of  jargon. 
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-  Increase  efforts  to  insure  that  evaluation  Information  is  used 
in  authorization  and  appropriations  processes  and  in  Federal  agency 
decisionmaking  processes. 
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DAY  I:  OVERVIEW  OF  RESPONSIBILITIES  AND  RESOURCES  FOR  LEGISLATIVE  OVERSIGHT 
AND  PROGRAM  EVALUATION:  SUMMARY  OF  PROCEEDINGS 


The  material  presented  at  the  first  day's  seminar  on  the  topic  of 
general  responsibilities  and  resources  for  legislative  oversight  and 
program  evaluation  set  the  theme  and  tone  for  succeeding  discussions. 
In  summary,  oversight  and  evaluation  do  not  consist  of  one  immutable 
process  applicable  to  all  legislative  decisionmaking  needs.   The  pur- 
poses, techniques,  and  resources  of  oversight  and  evaluation  differ  con- 
siderably depending  upon  a  decisionmaker's  needs,  and  related  to  these, 
tradeoffs  staff  have  to  make  among  available  evaluation  resources,  types 
of  programsbeing  assessed,  and  lead-time  available  for  information  gather- 
ing and  assessment.   However,  it  was  stressed  during  the  first  day's 
meeting  and  throughout  the  seminar  that  executive  and  legislative  staff 
sometimes  do  not  give  enough  attention  to  determining  how  particular  evalua- 
tion techniques  can  generate  information  appropriate  to  decisionmaking 
needs,  especially  with  respect  to  evaluations  mandated  by  the  Congress. 
This   lack  of  attention  can  result  in  needless  and  wasted  research  expenditu: 

The  first  day's  meeting  was  divided  into  two  parts:  first,  respon- 
sibilities and  needs  for  oversight  and  evaluation;  and  second,  resources 
for  oversight  and  evaluation. 

A.  Legislative  Responsibilities  and  Needs  for  Oversight  and  Evaluation 

Mr.  Lester  S.  Jayson,  Director  of  the  Congressional  Research  Service, 
opened  the  meeting  by  describing  the  importance  to  the  legislative  pro- 
cess of  an  exploratory  seminar  on  oversight  and  evaluation  .  Then  he 
described  the  agenda  for  the  series,  indicating  that  statements  would  be 
made  by  executive  branch  officials,  evaluation  contractors,  and  legisla- 
tive staff,  from  both  committee  and  congressional  information  and  analysis 
support  agencies. 
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1.  Scope  and  Limitations  of  Legislative  Oversight.   Walter  Oleszek , 
of  the  Government  Division  of  the  Congressional  Research  Service, 
presented  the  keynote  paper  on  the  first  part  of  the  seminar  theme, 
legislative  oversight.   He  defined  oversight  as  the  exceedingly  complex 
process  of  determining  whether  executive  agencies  carry  out  programs 

as  the  Congress  intended.   Three  types  of  oversight  can  be  distinguished: 
legislative,  fiscal,  and  investigative. 

Mr.  Oleszek  noted  that  the  Congress  has  encountered  innumerable 
problems  in  doing  oversight.   Chief  among  these  have  been  (1)  imprecise  def- 
inition of  programs  goals,  (2)  lack  of  adequate  staff,  (3)  lack  of  time 
required  to  do  adequate  oversight  (since  most  staff  time  is  used  to 
prepare  new  legislation),  (4)  non-objectivity  resulting  from  the 
mutually  reinforcing  relationships  that  sometimes  develop  between 
legislative  committees  and  Federal  agencies,  and  (5)  lack  of  established 
patterns  of  cooperation  among  authorizing,  appropriations,  and  government 
operations  committees  in  sharing  information  relevant  to  agency  and  program 
oversight. 

The  speaker  also  described  recent  legislative  activities  designed 
to  improve  committee  organization  and  information  use  and  gave  a  cautiously 
optimistic  assessment  of  their  implications  for  improving  legislative 
oversight.  These  actions  include:  the  Legislative  Reorganization  Act 
of  1970,  the  Congressional  Budget  and  Impoundment  Control  Act  of  1974, 
P.L.  93-344,  and  House  passage  of  the  Committee  Reform  Amendements  of 
1974,  H.Res.  988. 

2.  Types  of  program  Evaluation.   The  keynote  for  the  second  part 

of  the  seminar  theme  —  program  evaluation  was  given  by  Genevieve 

Knezo  of  the  Science  Policy  Research  Division,  Congressional  Research 
Service.   Ms.  Knezo  noted  the  increasing  demand  for  and  proliferation 
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of  evaluation  studies  as  well  as  the  growing    dissatisfaction  that 
members  of  Congress,  evaluation  researchers,  and  agency  administrators 
are  expressing  regarding  the  use  of  evaluation  in  policymaking,  includ- 
ing legislative  oversight.  Many  of  these  inadequacies,  she  noted,  might 
be  attributed  to  misunderstanding  by  legislative  staff  and  agency  admin- 
istrators about  the  techniques  used  and  products  generated  by  different 
types  of  evaluation. 

Two  types  of  evaluation  research  are  done  for  Federal  agencies: 
process  evaluations  and  impact  evaluations.   Process  evaluations;  are  used 
to  answer  management  questions;  generally  they  use  non-experimental 
evaluation  techniques  and  data  gathered  in  the  normal  course  of  adminis- 
stering  and  monitoring  a  program.   They  usually  are  not  as  costly  as  the 
second  type  of  evaluation:  impact  evaluation.   Impact  evaluations 
usually  answer  questions  about  the  impact  of  given  programs,  i.e.,  did  a 
program  cause  the  change  expected  in  attitude  or  behavior  in 
participants?  Impact  evaluations  are  difficult  to  do;  generally  they 
require  both  experimental  techniques  with  randomly  Selected  test  and  con- 
trol groups,  and  also  use  of  new  data  collection  techniques  which  have  to 
be  built  into  a  program  even  before  it  is  funded. 

Ms.  Knezo  also  introduced  a  theme  which  was  heard  several  times 

during  the  seminar  series:  sometimes  the  results  of  evaluation,  whether 

a  well-conducted  evaluation  or  a  poorly  conducted  evaluation,  or  whether 

favorable  or  unfavorable  to  a  program,  are  not  used  in  the  oversight 

process.   This  failure  occurs  because  the  legislative  process  requires 

that  decisions  be  based  on  objective  information  as  well  as  value 

preferences.   Sometimes  adoption  of  even  a  small-scale  experimental 

program  reflects  a  political  consensus  about  the  feasibility  and  c 

desirability  of  a  program  that  no  amount  of  evaluation,  whether  pro  or  con 
will  alter. 
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3.  The  Need  For  Dnderstandlng  Goals.   Dennis  Little, 
head  of  CRS  Futures  Group,   addressed  the  theme  of  the  need  for 
proper  specification  of  goals.   He  noted:  "In  order  to  conduct  a  mean- 
ingful agency  and  program  evaluation,  we  need  to  ask  the  right  questions 
at  the  right  time,  and  our  inquiry  must  take  place  at  several  levels. 
The  right  questions  can  be  asked  only  if  the  evaluator  understands  the 
agency's  goals,  missions,  program,  program  objectives,  and  program  and 
administrative  evaluation  criteria.   In  most  cases  such  an  understand- 
ing does  not  exist  and,  consequently,  many  evaluations  are  nonexistent 
or  of  questionable  utility." 

He  then  provided  a  framework  and  step-by-step  procedure  for  in- 
ducing agency  administrators  to  give  clear  statements  of  their  goals,  program 
objectives  in  support  of  these  goals,  information  needed  to  assess  achievement 
of  the  objectives  or  goals,  and  ways  to  determine  if  new  evaluation 
studies  were  indeed  required.   He  used  the  case  of  the  Social  Security 
Administration  as  an  example. 

B.  Legislative  Resources  for  Oversight  and  Evaluation 

The  second  part  of  the  first  afternoon's  presentations  consisted  of 
statements  about  legislative  resources  for  evaluation  and  oversight. 
Staff  of  the  Congressional  Research  Service,  the  General  Accounting  Of- 
fice, and  the  Office  of  Technology  Assessment  described  how  their  primary 
and  secondary  data  gathering  and  analysis  activities  could  serve  to  assist 
legislative  evaluation  users.   The  afternoon  concluded  with  statements 
from  staff  members  of  the  House  and  Senate  Government  Operations  commit- 
tees, describing  their  new  responsiblities . 
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1.  Resources  of  the  Congressional  Research  Service.   Dr .  Norman 
Beckman,  as  deputy  director  of  the  Congressional  Research  Service,  be- 
gan these  proceedings  by  providing  a  general  description  of  the  origin, 

evolution,  organization,  and  functions  of  the  Congressional  Research 
Service  as  a  legislative  support  agency.   He  described  several  types 
of  CRS  activities  which  assist  the  Congress  directly  in  performing  over- 
sight and  evaluation.   He  also  gave  examples  of  CRS  policy  analysis  re- 
ports which  contributed  directly  to  legislative  oversight  and  evaluation 
functions,  including  reports  relating  to  mineral  leasing;  redeployment  of 
U.S.  armed  forces  in  certain  countries;  implementation  of  revenue  shar- 
ing; community  development;  education  and  recreation  legislation; 
retirement  systems;  and  immigration. 

The  basic  authority  for  the  CRS  to  support  oversight  and  evaluation 
is  the  Legislative  Reorganization  Act  of  1970,  P.L.  91-510.   Several  of 
the  Act's  provisions  relate  specifically  to  oversight  and  evaluation.   For 
instance,  the  act  charges  the  Service  with  responsibility  for:  (1)  policy 
analysis  dealing  with  new  legislative  proposals  as  well  as  oversight 
programs;  (2)  new  anticipatory  functions,  including  the  submission  to 
each  committee  of  subject  and  terminating  program  lists  at  the  beginning 
of  each  new  Congress;  and  (3)  maintaining  continuous  liaison  with  all 
committees. 

Dr.  Beckman  also  detailed  innovative  directions  CRS  is  taking  to 
fulfill  its  responsibilities.   These  include  holding  public  policy  sem- 
inars for  members  and  staff,  which  permit  expert  speakers  to  discuss  all 
sides  of  an  issue,  and  improvement  and  enlargement  of  CRS  automated 
information  services.   These  information  systems  contain  data  bases 
directly  supportive  of  oversight  and  evaluation:  bill  digests  and  status 
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reports;  abstracted  bibliographic  information;  and  issue  briefs  which 
provide  in  a  few  pages  (in  print  or  on  videoscreen)  the  history,  current 
status  and  major  bibliographic  references  on  important  issues.   The 
automated  information  system  also  provides  researchers  with  access  to 
other  special  data  bases,  such  as  economic  models;  unpublished  economic, 
social,  and  educational  census  data;  special  Federal  agency  bibliographic 
data  bases,  such  as  ERIC,  MEDLARS,  and  MEDLINE:  and  legal  codes. 

a.   Emerging  issues  and  terminating  reports:   Dr.  Beckman  noted  that 
the  Legislative  Reorganization  Act  of  1970  gave  the  CRS  new  responsibili- 
ties for  assisting  committee  oversight  by  providing  committees,  at  the 
beginning  of  each  Congress,  with  lists  of  important  emerging  issues  and 
of  all  programs  and  activities  falling  under  each  committee's  juris- 
diction which  are  scheduled  to  terminate  in  that  Congress. 

David  E.  Gushee  of  the  Environmental  Policy  Division,  described 
the  procedures  CRS  uses  to  generate  and  transmit  such  lists  and  the 
form  the  lists  take.  A  prerequisite  for  doing  the  job,  he  noted,  is 
the  formation  of  interdivisional  CRS  task  forces  to  coordinate  the  prepara- 
tion of  emerging  issues  lists  for  committees  with  jurisdiction  over 
several  different  types  of  issues.   Preparation  of  the  emerging  issues 
lists  also  requires  special  current  awareness  by  CRS  analysts  and  good 
working  relationships  with  committee  staff.  He  reported  that  "For  the 
94th  Congress,  the  Service  organized  task  forces  for  49  committees.   The 
lists,  when  transmitted,  all  included  some  oversight  topics,  and  a  number 
of  them  included  agreement*  on  studies  the  CRS  would  either  do  or  partici- 
pate in  as  components  of  the  committee's  oversight  activities." 

For  the  94th  Congress,  the  Service  identified  421  expiring  programs 
over  half  were  authorizations  of  appropriations;  the  next  largest 
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categories  were  reports  from  agencies  and  commissions  and  "general 
authority." 

b.   Oversight  of  foreign  affairs  reports:  Often  legislation  con- 
tains language  requiring  Federal  agencies  to  report  annually  or  semi- 
annually to  the  Congress,  generating  information  to  be  used  for  over- 
sight purposes.   The  Foreign  Affairs  Division  of  the  Congressional 
Research  Service  assessed  the  utility  and  products  of  these  reporting 
requirements  in  the  foreign  affairs  field  jointly  for  the  Senate  Committee 
on  Foreign  Relations  and  the  House  Committee  on  International  Relations 
(formerly  called  the  Committee  on  Foreign  Affairs).   Mrs.  Ellen  Collier 
of  the  Foreign  Affairs  Division  described  the  work  in  this  area. 

The  first  report  done  for  the  committees  identified  several  in- 
adequacies of  reporting  requirements:  a  variety  of  different  and  hard- 
to-use  reporting  forms  were  used;  some  reports  lacked  dates  for  the 
reporting  period  covered,  or  date  of  publication;  in  several  cases 
statutory  requirements  led  to  overlapping  reporting  requirements;  many 
reports  were  submitted  too  late;  and  most  of  the  reports  were  superficial, 
glossy  public  relations  documents  inferior  in  quality  to   informational 
or  analytical  reports  needed  for  good  oversight. 

In  its  first  report  to  the  Committees,  the  Foreign  Affairs  Division 
observed  that  legislative  use  of  reports  for  evaluation  and  oversight 
would  be  enhanced  if  steps  were  taken  to  improve  the  quality  of  reports 
and  if  a  central  information  receiving  and  monitoring  system  were 
established. 

About  a  year  later  the  Division  prepared  a  second  report  for  the 
committees  evaluating  Federal  agency  responses  to  the  recommendations 
of  the  first  report.   Although  the  Division  found  improvements,  which 
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are  generating  better  information  for  the  committees,  it  noted  that  far 
more  work  is  needed  to  improve  reporting  requirements  and  compliance  with 
them.   At  the  request  of  the  two  committees,  the  Division  is  now  monitoring 
and  analyzing  several  hundred  reports  which  are  submitted  to  the  Congress 
by  executive  branch  agencies  each  year  in  accordance  with  about  350 
separate  statutory  requirements  in  foreign  affairs  legislation. 

2.  Resources  of  the  General  Accounting  Office.   James  P.  Oliver, 
of  the  General  Accounting  Office,  made  the  next  presentation.   He  gave 
a  succinct  statement,  illustrated  with  numerous  charts  and  graphs,  ex- 
plaining new  legislative  budgeting  requirements  mandated  by  the  Congres- 
sional Budget  and  Impoundment  Control  Act  of  1974,  P.L.  93-344.   He  noted, 
in  particular,  that  properly  timed  evaluation  was  needed  to  fulfill  this 
act  and  to  assist  authorizing  committees  in  justifying  program  requests 
as  required  under  the  act.   He  also  briefly  described  the  relationships 
between  the  Office  of  Management  and  Budget,  the  Congressional  Budget 
Office,  the  General  Accounting  Office,  and  the  House  and  Senate  Budget 
Committees  in  fulfilling  the  provisions  of  the  law. 

The  General  Accounting  Office     plans  to  establish  a  special 
Congressional  Information  Services  Group  to  enable  the  agency  to  fulfill 
its  responsibilities  under  the  act  for  improving  and  coordinating  program 
information  and  program  evaluation  techniques  and  reports .   The  group  has 
responsibility  for  the  following  tasks:  development  of  information 
structures  and  evaluation  methods;  improvement  of  reporting  to  the  Con- 
gress of  information  and  evaluations;  cooperation  with  the  executive 
branch  in  improving  information  systems  and  evaluation  processes;  improve- 
ment of  congressional  and  GAO  capabilities  to  acquire  and  use  informa- 
tion and  programs  evaluations;  and  cooperation  with  States  and  local 
governments  in  improving  the  flow  of  evaluation  information. 


3.  Resources  of  the  office  of  Technology  Assessment.   Dr.  Gary 
Thomas,  a  staff  member  of  the  Office  of  Technology  Assessment  working 
on  materials  assessments,  presented  the  next  paper,  which  placed  the 
OTA  assessment  activities  in  the  context  of  congressional  oversight  and 
evaluation  responsibilities.   The  speaker  described  the  origins  and  current 
projects  being  undertaken  by  the  OTA  at  the  request  of  the  Congress. 

Dr.  Thomas  also  discussed  two  issues  of  special  importance  to  the 
oversight  and  evaluation        role  played  by  a  legislative  support 
agency.   He  noted  that  both  its  legislative  mandate  and  legislative 
exigencies  require  the  OTA's  work  to  be  especially  timely  and  responsive 
to  the  Congress.   The  requirement  is  designed,  in  part,  to  overcome  an 
inadequacy  described  throughout  the  meeting:  that  evaluation  reports 
done  by  Federal  agencies  and  the  GAO  frequently  are  of  little  use  to 
legislative  decisionmaking  needs  because  they  are  too  late. 

In  this  same  respect,  Dr.  Thomas  noted  that  the  OTA  does  long-  as 
well  as  short-range  assessments.   Short  range  assessments  consist  of 

two  types  those  done  to  explore  the  feasibility  and  need  for  a  more 

in-depth  study  and  those  done  on  an  urgent  basis  for  a  particular  com- 
mittee. 

Dr.   Thomas  also  noted  that  OTA  maintains  very  close  liaison  with 

committees  both  to  receive  suggestions  for  assessments,  and  to  brief 

committee  staff  throughout  the  process  of  the  assessent,  making  avail- 
able working  documents  and  monthly  reports  contractors  submit. 

4.  Oversight  and  Evaluation  in  the  Senate  Government  Operations 
Committee.       The  next  set  of  presentations  consisted  of  discussions 
by  staff  members  of  the  Senate  and  Hou6e  Government  Operations  Committees 
describing  their  responsibilities  for  oversight  and  evaluation. 
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Ms.  Marilyn  Harris,  chief  clerk  of  the  Senate  Committee  on  Govern- 
ment Operations, led  the  discussion  by  committee  staff  members.   She 
noted  that  the  committee  has  oversight  responsibilities  for  all  Federal 
agencies.   To  fulfill  these  responsibilities  adequately,  the  Committee 
recently  reorganized,  creating  a  functionally  and  financially  autonomous 
subcommittee  structure.   She  also  described  the  effectiveness  of  informa- 
tion exchange  and  cooperation  among  the  subcommittees,  and  between  the 
Government  Operations  Committee  and  other  standing  committees  with 
legislative  authority. 

The  Permanent  Subcommittee  on  Investigations,  which  does  not  have 
legislative  authority,  conducts  investigations  on  topics  ranging  from 
crime  to  the  Russian  grain  sale.   Howard  Feldman,  Subcommittee  General 
Counsel,  noted  two  especially  important  oversight  information  gathering 
tools:  the  use  of  the  subpena  power  and  of  highly  trained  professional  in- 
vestigators.  He  also  noted  that  the  reports  of  the  Subcommittee  usually 
lead  to  the  initiation  of  legislative  action  by  other  committees  with 
legislative  jurisdiction  for  the  issue. 

The  Intergovernmental  Relations  Subcommittee  has  legislative  over- 
sight responsibilities  for  the  relationships  between  Federal,  State, 
and  local  government  officials  in  areas  dealing,  for  instance,  with 
Federal  aid  programs,  questions  of  constitutional  authority,  and  the 
Freedom  of  Information  Act.  The  Subcommittee  Counsel,  Jim  Davidson, 
described  investigations  dealing  with  revenue  sharing,  the  impact  of 
the  energy  crisis  on  State  and  local  government,  and  the  New  Federalism. 
This  Subcommittee  had  made  effective  use  of  another  important  oversight 
information  gathering  tool:  the  use  of  a  Harris  public  opinion  poll  to 
obtain  citizen's  views  about  the  performance  of  State  and  local 


38 


govermental  officials.   Current  oversight  activities  include  implementa- 
tion by  Federal  agencies  of  the  Freedom  of  Information  Act  and  of  activi- 
ties of  U.S.  intelligence  gathering  agencies. 

A  new  subcommittee,  the  Oversight  Procedures  Subcommittee,  was  created 
during  the  94th  Congress.   Its  general  objectives  are  to  assist  other  Senate 
committees  in  improving  the  oversight  function  and  to  anticipate  problems  re- 
quiring investigation  before  they  become  major  issues.   Its  four  specific 
responsibilities  are:  (1)  to  help  formulate  better  methods  and  procedures 
for  oversight;  (2)  to  oversee  the  use  and  disposal  of  Federal  property; 
(3)  to  assist  in  developing  new  budget  evaluation  measures;  and  (A)  to 
oversee  the  work  of  the  Paperwork  Commission. 

Bill  Goodwin,  Subcommittee  Staff  Director,  gave  several  examples  of 
current  activities  to  improve  oversight  and  evaluation.  These  include  study- 
ing the  Senate  committee  structure  to  consolidate  jurisdictional  responsibili- 
ties and  working  with  the  Congressional  Research  Service  and  the  General 
Accounting  Office  to  prepare  and  publish-  materials  on  oversight  and  evalua- 
tion procedures  and  resources. 

The  Staff  Director  of        the  Reports,  Accounting,  and  Management 
Subcommittee,  Vic  Reinemer,  gave  details  on  the  need  for  and  procedures 
used  in  two  current  oversight  areas.   First  is  an  investigation  of  over  - 
legislation.   The  committee  believes  that  the  legislative  process  would 
be  improved  considerably  if  efforts  were  made  to  review  and  revise  existing 
statutes  rather  than  adopting  new  legislation  which  may  duplicate  or  counter- 
act existing  laws.  The  Subcommittee  is  using  the  services  of  the  General 
Accounting  Office  and  the  Congressional  Research  Service  to  carry  out 
this  investigation. 
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The  second  area  of  investigation  is  oversight  of  the  Federal  Ad- 
visory Committee  Act.   The  committee's  preliminary  research  indicates 
that  the  Congress  and  executive  agencies  are  creating  far  too  many  ad- 
visory committees;  and  that  existing  advisory  committees  can  assume 
whatever  new  advisory  functions  may  be  required.   The  Congressional 
Research  Service  and  the  General  Accounting  Office  are  also  being  used 
for   data  gathering  and  analysis  on  this  oversight  question. 

The  next  speaker  was  a  Counsel  to     the  Subcommittee  on  Federal 
Spending  Practices,  Efficiency,  and  Open  Government,  Ron  Chiodo.   The 
Subcommittee's  jurisdiction  extends  to  Federal  procurement  laws  and 
regulations,  efficiency,  and  economy  of  Federal  spending  practices 
generally,  and  the  open  conduct  of  government  meetings,  including  those 
f  the  Coneress  Mr"  Cniodo  8ave  details  on  the  objectives  and  procedures  used  for 

several  current  oversight  investigations,  including:  cost  overruns  of 
military  R  and  D,  due  to  lack  of  competitive  procurement  procedures;  the 
need  to  consolidate  Federal  procurement  regulations;  oversight  of  pro- 
cedures to  review  and  award  grants  and  contracts  in  the  Department  of 
Health,  Education,  and  Welfare;  and  "Sunshine"  legislation. 

5.  Oversight  responsibilities  of  the  House  Government  Operations 
Committee.   Under  the  provisions  of  H.  Res.  988,  the  Committee  Reform 
Amendments  of  1974,  each  committee  and  subcommittee  of  the  House  is  re- 
quired to  provide  the  House  Committee  on  Government  Operations  with  a 
plan  for  its  oversight  activities  at  the  beginning  of  each  Congress. 
The  objective  of  this  requirement  is  to  enhance  oversight  of  Federal 
programs  which  fall  under  the  Committee's  jurisdication  and  to  attempt 
to  coordinate  oversight  activities  of  the  House  committees.   The 
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Government  Operations  Committee  also  has  special  investigative  over- 
sight responsibilities. 

Bill  Jones,  Staff  Director  of  the  Committee,  described  the 
purposes,  methods  used,  and  expected  utility  to  the  House  of  the  over- 
sight coordination  work  the  committee  did  at  the  beginning  of  the  94th 
Congress.   Committee  staff,  from  both  the  majority  and  minority  sides 
polled  members  and  staff  of  all  House  committees  to  obtain  information 
on  the  committee's  oversight  plans.   These  were  later  published  in  a 
committee  print  which  is  included  as  an  appendix  to  this  report. 

Mr.  Jones  said  that  the  document's  major  purpose  is  to  disseminate 
information  about  oversight  responsibilities  and  to  prompt  committees 
to  develop  long-range  oversight  plans.   There  are  evidences  of  overlap 
in  oversight  jurisdiction;  however,  according  to  Mr.  Jones,  the  Govern- 
ment Operations  Committee  does  not  now  have  a  responsibility  to  force 
resolution  of  overlaping  jurisdictions.  Instead  the  committee  encourages 
the  separate  committees  and  subcommittees  involved  to  resolve  differences 
or  hold  joint  investigations  or  hearings. 
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'INTRODUCTORY  REMARKS",  MR.  LESTER  S.  JAYSON, 
DIRECTOR,  CONGRESSIONAL  RESEARCH  SERVICE 


I  welcome  you  today  to  the  first  of  three  afternoon  sessions  in  a 
seminar  series  on  legislative  oversight  and  program  evaluation.   The  se- 
ries is  being  sponsored  by  the  Congressional  Research  Service  at  the  re- 
quest of  Senator  Lee  Metcalf,  Chairman  of  the  Senate  Government  Operations 
Subcommittee  on  Reports,  Accounting,  and  Management,  and  Senator  Bill 
Brock,  Ranking  Minority  Member  of  that  subcommittee.   The  purpose  of  the 
seminars  is  to  acquaint  legislative  staff  who  have  responsibilities  or 
interest  in  legislative  oversight  and  program  evaluation  with  current 
requirements,  processes,  and  resources. 

The  seminars  are  designed  to  bring  together  useful  information  on 
the  various  resources  available  to  Congress  or  assistance  in  the  over- 
sight and  evaluation  process,  and  to  present  a  variety  of  actual  experi- 
ences with  oversight  and  evaluation  in  both  the  Congress  and  the  Executive. 
The  sessions  are  meant  to  be  informal  and  exploratory  rather  than  an  ex- 
haustive inventory  or  assessment  of  legislative  oversight  responsibilities 
and  techniques.   Active  participation  by  the  audience  is  essential  to  the 
success  of  this  series,  and  I  encourage  you  to  raise  questions  and  offer 
constructive  comments  in  the  discussion  period  that  follows  each  presen- 
tation. 

The  speakers  for  today's  session  will  outline  new  legislative  over-- 
sight  and  evaluation  responsibilities  and  the  conceptual  and  methodolo- 
gical links  between  these  two  functions.   Staff  from  three  organizations 
which  have  legislative  support  functions  —  CRS,  GAO,  and  OTA  —  will 
describe  some  of  their  resources  and  activities  in  support  of  oversight 
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and  evaluation,  using  case  studies  as  examples  where  appropriate.   Repre- 
sentatives of  the  House  and  Senate  Government  Operations  Committees  will 
then  outline  their  perspectives  on  committee  oversight  responsibilities 
and  describe  related  activities. 

On  the  second  day,  May  12,  four  committee  case  studies  representing 
different  substantive  areas  and  approaches  to  legislative  oversight  and 
evaluation  are  scheduled  for  presentation.   These  case  studies  should 
help  illustrate  how  different  types  of  evaluation  strategies  are  required 
for  effective  oversight  and  assessment  of  different  program  areas.   Each 
of  these  strategies,  presumably,  has  distinct  advantages  and  limitations  — 
conceptual,  methodological,  and  political  —  that  must  be  taken  into  con- 
sideration. 

The  third  session  on  May  19  is  designed  to  explore  executive  branch 
experience  with  program  evaluation  and  the  evaluation  process  itself. 
The  General  Accounting  Office  and  the  authors  of  a  program  evaluation  man- 
ual for  legislative  staff  will  discuss  evaluation  processes  and  appropri- 
ate legislative  language  and  techniques  in  relation  to  legislative  program 
goals.   The  Office  of  Management  and  Budget  will  describe  the  steps  being 
taken  by  OMB  to  improve  the  conduct  and  use  of  evaluation  in  the  executive 
branch.   Dr.  Clark  Abt,  who  has  extensive  experience  directing  federally 
supported  program  evaluation  research  studies,  will  give  an  overview  of 
the  state  of  the  art  of  evaluation  techniques  and  an  assessment  of  the 
strengths  and  weaknesses  of  different  approaches.   Dr.  John  Evans  of  the 
Office  of  Education  will  discuss  practical  issues  and  problems  with  re- 
spect to  evaluation  of  education  programs,  particularly  as  they  relate  to 
the  legislative  process  and  to  congressional  interests. 
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A  more  detailed  outline  of  the  second  and  third  seminar  sessions  is 
available  for  your  information  and  for  circulation  to  others  you  think 
may  be  interested. 

In  addition,  relevant  CRS  and  other  background  materials  will  be  on 
the  table  by  the  door  at  each  session  for  your  information  and  use.   These 
include  bibliographies  on  evaluation,  a  report  on  emerging  issues  relating 
to  evaluation,  and  the  House  Government  Operations  Committee  report  on 
the  oversight  plans  of  the  House  committees. 

Since  I  will  not  be  able  to  be  present  at  all  sessions,  I  have  asked 
Norman  Beckman,  Deputy  Director  of  CRS,  to  be  the  moderator  of  the  sem- 
inar series. 

Now  I  would  like  to  introduce  the  first  group  of  speakers:  Mr.  Walter 
Oleszek  of  our  Government  Division,  Mr.  Dennis  Little  of  the  CRS  Futures 
Group,  and  Ms.  Jean  Knezo  of  the  Science  Policy  Research  Division. 
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'CONGRESSIONAL  OVERSIGHT:   A  REVIEW  OF  RECENT  LEGISLATIVE  ACTIVITIES," 
WALTER  J.  OLESZEK,  CONGRESSIONAL  RESEARCH  SERVICE 


While  Members  and  scholars  have  long  stated  that  oversight  of 
Federal  agencies  and  programs  is  a  principal  and  traditional  function 
of  Congress,  it  was  not  until  1946  that  Section  136  of  the  Legisla- 
tive Reorganization  Act  (60  Stat.  132)  formally  required  Congress's 
standing  committees  to  exercise  "continuous  watchfulness"  of  the 
execution  of  laws  by  administrative  agencies  under  their  jurisdic- 
tion. While  always  considered  an  important  function,  Members  today 
exhibit  heightened  concern  about  oversight.   Before  discussing  why 
there  is  increased  concern  with  oversight,  it  might  be  helpful  to 
define  the  term. 

What  is  Oversight? 

The  1946  Legislative  Reorganization  Act  divided  Congress'  over- 
sight function  among  three  types  of  standing  committees.   First,  author- 
izing committees  (Agriculture,  Veterans'  Affairs,  Commerce,  and  the 
like)  principally  had  "legislative"  oversight  over  programs  and  agen- 
cies under  their  jurisdiction.   These  committees  were  to  assess  whether 
programs  were  working  or  not,  and  propose  legislative  remedies  to  re- 
solve whatever  program  deficiencies  they  might  have  uncovered. 

"Fiscal"  oversight  was  the  primary  concern  of  the  Appropriations 
Committees  of  each  chamber,  which  were  to  scrutinize  the  spending  of 
funds  by  Federal  agencies.   Finally,  the  1946  LRA  granted  wide-ranging 
"investigative"  oversight  to  the  House  and  Senate  Committee  on  Govern- 
ment Operations.   With  their  authority  to  study  the  "operation  of  Gov- 
ernment activities  at  all  levels  with  a  view  to  determining  its 
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economy  and  efficiency,"  the  Government  Operations  Committees  were 
authorized  to  examine  the  administration  and  execution  of  programs 
that  cut  across  both  agency  and  committee  jurisdictions,  uncovering 
waste,  mismanagement,  and  abuses  of  authority.   None  of  these  types 
of  oversight,  it  should  be  stressed,  is  mutually  exclusive  of  the 
others. 

In  performing  these  different  types  of  oversight,  Congress,  its 
committees,  and  individual  members  have  available  a  range  of  over- 
sight techniques  to  insure  that  (1)  program  objectives  established 
by  Congress  are  being  met,  (2)  executive  waste  and  dishonesty  are 
eliminated,  (3)  agencies  are  properly  administered,  and,  (A)  execu- 
tive policies  reflect  the  public  interest.   Some  of  these  techniques 
include:   "legislative"  or  "committee"  vetoes,  casework,  hearings, 
and  investigations,  statutory  controls,  investigations  by  the  Gen- 
eral Accounting  Office,  and  the  establishment  of  reporting  require- 
ments for  Federal  agencies. 

While  the  range  of  oversight  techniques  is  broad,  there  is  rec- 
ognition by  many  legislators  that  Congress  is  deficient  in  the  con- 
duct of  oversight.   During  1973  hearings  before  the  House  Select 
Committee  on  Committees,  Member  after  Member  testified  as  to  the 
inadequacies  of  committee  oversight.   As  one  Representative  said, 
"I  have  never  felt  that  any  committee  was  doing  an  adequate  job  on 
oversight,  and  I  felt  very  strongly  our  committee  was  not  doing  a 
good  job  on  oversight  through  the  years."   33/   To  strengthen  that 


33/   Committee  Organization  in  the  House.   Hearings  before  the  House 
Select  Committee  on  Committees,  93rd  Congress,  1st  session,  p. 
310. 
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function,  many  oversight  reforms  have  been  adopted  in  recent  years 
(see  discussion  infra) .   Several  possible  factors  explain  the  in- 
creased concern  with  oversight,  and  why  Members  have  problems  con- 
ducting systematic  review  of  Federal  programs  and  agencies. 

Factors  Prompting  Concern  With  Oversight 

The  events  associated  with  Watergate  have  spotlighted  the  im- 
portance of  oversight.   Some  observers  have  specifically  concluded 
that  if  congressional  committees  had  been  more  diligent  in  over- 
seeing such  agencies  as  the  FBI,  CIA,  and  IRS,  many  of  the  Watergate 
abuses  might  never  have  occurred.   For  example,  the  National  Academy 
of  Public  Administration  said  in  March  1974  that  "continuing  over- 
sight could  have  made  Members  of  Congress  aware  much  earlier  of  the 
centralization  of  power  occurring  within  the  Executive  through  the 
conversion  of  presidential  advisers  to  'assistant  Presidents.'"  The 
Academy  recommended  that  Congress  "give  major  attention  to  strength- 
ening its  capacity  to  perform  the  oversight  function,  with  particular 
emphasis  on  evaluating  the  performance  of  executive  agencies."   34/ 
Oversight  might  also  help  rebuild  public  trust  in  the  Federal  Gov- 
ernment. 

Second,  the  1973-1974  oil  embargo  focused  congressional  and  pub- 
lic attention  on  the  false  assumption  that  the  world  capacity  to  pro- 
duce oil  was  in  no  danger  and  sufficient  to  support  U.S.  needs.  Many 


34/  Watergate:   Its  Implications  for  Responsible  Government.   A  re- 
port prepared  by  a  Panel  of  the  National  Academy  of  Public  Ad- 
ministration at  the  request  of  the  Senate  Select  Committee  on 
Presidential  Campaign  Activities,  March  1974,  pp.  71-72. 
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are  now  aware  that  the  United  States,  at  least  in  certain  areas,  may 
be  moving  from  an  economy  of  abundance  to  one  of  scarcity.   Hence, 
legislators  are  more  sensitive  than  ever  to  the  need  to  eliminate 
waste  and  mismanagement  in  program  administration,  to  make  every  dol- 
lar count,  to  authorize  programs  that  will  work,  and  to  terminate  or 
reduce  the  funding  of  those  programs' which  do  not  function  properly. 

Third,  particularly  since  the  1960s  (the  "Great  Society"  in- 
novations for  example)  the  number  of  Federal  programs  has  prolifer- 
ated.  From  the  small  businessman  anxious  to  receive  a  loan  from  the 
Small  Business  Administration  to  the  elderly  person  dependent  on  Med- 
icare payments,  more  and  more  citizens  are  affected  by  Federal  pro- 
grams. And  if  things  go  wrong  in  those  programs,  then  citizens  will 
alert  their  Representative  or  Senator.   Moreover,  there  are  today  per- 
haps more  groups  and  associations,  than  in  the  past,  which  monitor  the 
implementation  of  Federal  programs.   As  an  example,  the  Brookings  In- 
stitution has  recently  published  a  study  on  Monitoring  Revenue  Sharing, 
which  examines  the  general  revenue  sharing  program,  which  is  to  be  re- 
considered by  the  94th  Congress.   35/ 

Finally,  there  are  numerous  Representatives  and  Senators  who  are 
interested  in  strengthening  Congress'  oversight  function.   This  is 
vital  to  increased  oversight  because  even  if  there  is  a  propitious 
oversight  climate,  legislators  and  outside  groups  must  still  capi- 
talize upon  those  political  forces  if  oversight  is  to  be  strengthened. 


35/  Richard  P.  Nathan,  Allen  D.  Manvel,  Susannah  E.  Calkins  and  As- 
sociates, Monitoring  Revenue  Sharing  (Washington,  D.C.:   The 
Brookings  Institution,  1975). 
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Many  Democrats  and  Republicans,  conservatives  and  liberals,  support 
improvements  in  oversight,  as  witness  Democratic  liberal  John  Culver 
(Iowa)  and  conservative  Republican  Dave  Martin's  (Neb.)  strong  sup- 
port for  the  creation  of  oversight  subcommittees  on  all  House  commit- 
tees. 

Problems  in  Conducting  Oversight 

Many  people  on  Capitol  Hill  assume  that  systematic,  comprehensive 
review  of  programs  would  be  attainable  if  only  Members  were  more  dili- 
gent.  But,  as  Professor  Morris  Ogul  has  stated:   36/ 

No  amount  of  congressional  dedication  and  energy,  no  conceivable 
increase  in  the  size  of  committee  staffs,  and  even  no  extraordinary 
boost  in  committee  budgets  will  enable  the  Congress  to  carry  out 
its  oversight  obligations  in  a  comprehensive  and  systematic  manner. 
The  job  is  too  large  for  any  combination  of  members  and  staff  to 
completely  master.   Congressmen  who  feel  obligated  to  obey  the  let- 
ter of  the  law  are  doomed  to  feelings  of  inadequacy  and  frustration 
and  are  laid  open  to  charges  of  neglect.   Fortunately,  at  least  for 
their  own  morale,  the  members  of  the  Congress  tend  to  focus  on  the 
immediate  more  than  the  impossible. 

Nevertheless,  there  are  committees  that  lack  sufficient  staff  to  con- 
duct meaningful  review  work  and  (or)  neglect  to  establish  an  oversight 
agenda  determining  which  programs  and  agencies  under  their  jurisdiction 
have  review  priority. 

Most  of  the  time,  oversight  is  performed  selectively,  even  er- 
ratically, because  the  costs  of  conducting  systematic  and  comprehen- 
sive review  are  generally  perceived  to  be  too  high.   Many  Members  sim- 
ply lack  the  time  (or  interest)  to  explore  the  numerous,  and  often 


36/  Morris  S.  Ogul.   "Legislative  Oversight  of  Bureaucracy,"  in  Com- 
mittee Organization  in  the  House,  Panel  discussions  before  the 
House  Select  Committee  on  Committees,  93rd  Congress,  1st  session, 
p.  702. 
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complicated,  facets  of  programs.   This  is  often  tedious  work  with  no 
guarantee  that  Members  will  reap  any  political  reward,  either  within 
their  own  chamber  or  among  their  outside  constituents.   Moreover,  many 
committees  confront  a  heavy  legislative  work  load  of  bills  proposing 
new  programs  or  expansion  of  old  ones  and  must  of  necessity  devote  al- 
most full-time  attention  to  the  enactment  of  these  measures.   As  a  re- 
sult, little  time  is  left  for  oversight. 

Some  committee  members  have  a  narrow  conception  of  when  oversight 
should  be  done.   As  one  chairman  recently  stated,  "I  am  ready  to  do  it 
(oversight)  whenever  there  is  evidence,  but  I  want  indictment  first."  37/ 
Of  course,  one  can  argue  that  oversight  is  only  necessary  when  an  agen- 
cy's performance  is  found  wanting  and  that  committee-agency  consensus 
on  policy  matters  indicates  little  need  for  systematic  review.   It  is 
not  unusual,  however,  for  committees  to  be  "stacked"  with  members  who 
share  similar  backgrounds  and  the  values  espoused  by  agencies.   This 
encourages  committee-agency  consensus.   Such  consensus  is  a  strong  in- 
dication of  the  need  for  legislative  review  of  that  agency. 

Oversight  can  also  be  inhibited  by  the  alliance  that  often  de- 
velops between  committees  and  agencies.   Mutually  supportive  relation- 
ships can  develop  between  the  committees  that  authorize  various  pro- 
grams and  the  agencies  that  administer  them.   Of  course,  a  committee 
may  simply  agree  with  a  policy  and  see  no  need  for  oversight,  or  it 
may  trust  a  Federal  official  so  much  that  the  committee  sees  no  need 
to  review  the  agency  he  heads.   Moreover,  as  authors  of  legislative 
policy,  committees  may  want  program  administration  to  "look  good." 


37/   Committee  Organization  in  the  House.   Hearings,  p.  67. 


50 


The  Legislative  Reorganization  Act  of  1946  attempted  to  deal  with 
these  situations  through  the  broad  investigative  authority  granted  the 
House  and  Senate  Committees  on  Government  Operations.  Not  tied  to  any 
single  Federal  agency  or  program,  these  committees  can  investigate  pro- 
grams and  agencies  that  are  within  the  jurisdiction  of  another  commit- 
tee. However,  the  Government  Operations  Committees  have  not  been  as 
effective  as  many  Members  and  scholars  had  hoped. 

Among  problems  the  Government  Operations  Committees  have  encoun- 
tered are  synthesizing  and  transmitting  information  in  a  way  useful 
for  other  committees  and  achieving  some  degree  of  coordination  and 
cooperation  in  the  sharing  of  information  among  the  different  types 
of  committees. 

Another  factor  limiting  oversight  is  the  lack  of  clarity  regard- 
ing legislative  intent.   Public  laws  are  often  the  product  of  con- 
flicts and  compromises,  and  when  those  compromises  are  translated  into 
legislative  language,  it  may  not  be  possible  to  determine  with  any  de- 
gree of  assurance  what  specifically  was  intended  for  program  adminis- 
tration.  And  on  controversial  bills,  there  may  not  even  be  any  clear 
consensus  regarding  the  goals  or  objectives  of  a  program  by  committee 
members  of  staff.   Hence,  this  sometimes  leads  to  internal  committee 
bickering  about  what  constitutes  clear  standards  for  indices  of  pro- 
gram success  or  failure.   Section  702  of  the  Congressional  Budget  and 
Impoundment  Control  Act  (P.L,  93-344)  recognizes  this  problem  and  di- 
rects the  General  Accounting  Office  to  assist  committees  in  developing 
statements  of  legislative  objectives  in  order  to  facilitate  program 
evaluation. 
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The  way  a  committee  functions  can  also  determine  the  extent  of 
oversight.   Committees  organized  with  rather  independent  subcommit- 
tees, adequately  staffed  and  funded,  are  more  likely  to  pursue  over- 
sight than  are  committees  where  a  strong  chairman  dominates  his  col- 
leagues.  With  established  jurisdictions,  subcommittees  are  better 
able  to  conduct  realistic  and  manageable  reviews  of  programs  and 
agencies. 

Recent  Oversight  Reforms 

1.  Legislative  Reorganization  Act  of  1970  (P.L.  91-510).   The 
1970  act  requires  each  standing  committee  to  submit  a  biennial  report 
on  its  legislative  review  activities;  assigns  new  program  review  re- 
sponsibilities to  the  General  Accounting  Office;  and  directs  the  Con- 
gressional Research  Service  to  provide  greater  substantive  and  policy 
analysis  support  to  congressional  committees. 

2.  Congressional  Budget  and  Impoundment  Control  Act  (P.L.  93-344). 
Title  VII  of  the  Budget  Act  is  entitled  "Program  Review  and  Evaluation." 
The  act  authorizes  committees  to  fulfill  their  oversight  responsibilities 
by  contract  or  by  requiring  Federal  agencies  to  conduct  evaluations.  Com- 
mittees may  also  use  evaluative  techniques  to  pilot  test  programs  or  to 
do  cost-benefit  analyses.   In  addition,  the  General  Accounting  Office  is 
directed  to  assist  committees  in  developing  statements  of  legislative 
objectives,  and  to  establish  an  Office  of  Program  Review  and  Evaluation 
to  carry  out  the  mission  described  in  Title  VII. 

3.  Committee  Reform  Amendments  of  1974  (H.  Res.  988).   House 
Resolution  988,  the  Committee  Reform  Amendments,  sought  to  strengthen 
the  oversight  role  of  the  House  Committee  on  Government  Operations  in 
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particular  and  that  of  other  committees  in  general.   The  Government 
Operations  Committee  has  been  assigned  the  following  new  responsibili- 
ties: 

Preparation  of  an  Oversight  Report:   Within  60  days  after  a  new 
Congress  convenes,  the  committee  is  to  report  to  the  House  on  the 
oversight  plans  of  every  committee  and  assist  in  coordinating  the 
oversight  activities  of  the  House.   The  purpose  of  this  proposal  is 
to  help  facilitate  coordination  and  cooperation  among  committees  on 
oversight  matters  and  to  inform  the  House  as  to  the  oversight  plans 
of  the  committees. 

Expanded  Investigative  Authority:   H.  Res.  988  made  clear  that 
the  Committee  on  Government  Operations  need  not  refrain  from  investi- 
gating a  subject  area  because  another  committee  is  studying,  or  has 
the  jurisdiction  to  study,  the  same  area.   Before,  the  committee  was 
often  reluctant  to  investigate  matters  being  considered  by  another 
committee. 

Summary  of  Review  Findings  to  be  Considered  by  Other  Legislative 
Committees:   Very  often,  the  investigative  findings  of  the  Committee 
on  Government  Operations  were  not  considered  by  the  other  standing 
committees  during  their  legislative  deliberations.   H.  Res.  988  makes 
clear  that  the  pertinent  review  findings  and  recommendations  of  Gov- 
ernment Operations  are  to  be  considered  by  the  authorizing  committees, 
if  presented  to  it  in  a  timely  fashion.   Moreover,  the  report  of  the 
authorizing  committee  must  include  the  findings  and  recommendations  of 
Government  Operations  in  a  section  of  the  report  that  is  separately  set 
out  and  clearly  identified. 
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Oversight  Findings  Noted  on  Conmittee  Reports:   Committees  are 
now  required  to  indicate  on  the  cover  of  their  reports  that  they  con- 
tain a  summary  of  the  oversight  findings  of  the  Committee  on  Govern- 
ment Operations  when  that  is  the  case. 

All  of  the  above  changes  were  designed  to  strengthen  the  over- 
sight role  of  Government  Operations  and  to  reduce  the  isolation  of 
committees.   Very  often  authorizing  committees  are  not  aware  of  what 
Government  Operations  is  doing  and  vice  versa.   Hence,  neither  prof- 
its from  the  work  of  the  other.   The  changes  are  designed  to  elimi- 
nate that  condition  and  to  help  promote  cooperation  and  coordination 
among  the  committees. 

H.  Res.  988  required  other  oversight.   First,  all  committees  now 
have  the  authority  to  review  the  impact  of  tax  expenditures  on  matters 
that  fall  within  their  jurisdiction.   Tax  expenditures  refer  to  money 
not  collected  because  of  various  "loopholes"  (tax  subsidies,  incentives, 
or  preferences)  in  the  tax  code,  which  will  total  in  fiscal  year  1976 
nearly  $92  billion.   By  reviewing  tax  expenditures  within  their  juris- 
dictions, each  committee  will  have  a  more  complete  picture  of  whether 
the  financial  assistance  being  made  available  to  or  through  various 
programs  is  achieving  national  objectives. 

Second,  each  committee  (except  Appropriations  and  Budget)  has  a 
futures  research  and  forecasting  responsibility.  On  the  theory  that 
what  was  will  influence  what  might  be,  then  oversight  and  foresight 
are  related.  Committees  now  have  the  responsibility  to  identify  and 
assess  conditions  or  trends  that  might  require  future  legislative  ac- 
tion. Hence,  the  House  may  better  be  able  to  anticipate  problems  be- 
fore they  become  crises. 
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Third,  the  Select  Committee  developed  the  concept  of  "special  over- 
sight," and  H.  lies.  988,  as  adopted,  granted  seven  standing  committees 
this  responsibility.   Issues  are  simply  too  complex  to  divide  into  water- 
tight compartments.   Special  oversight  insures  that  certain  committees 
have  the  right  to  conduct  comprehensive  oversight  by  enabling  them  to 
oversee  programs  and  agencies  that  fall  within  the  jurisdiction  of  an- 
other committee. 

Fourth,  H.  Res.  988  authorized  committees  to  establish  oversight 
subcommittees  or  to  require  their  subcommittees,  if  any,  to  conduct 
oversight  in  their  jurisdictional  areas.  At  least  nine  House  commit- 
tees have  established  oversight  entities. 

Fifth,  committee  reports  on  measures  shall  include  their  over- 
sight findings  separately  set  out  and  clearly  identified. 

Finally,  the  costs  of  stenographic  services  and  transcripts  for 
oversight  hearings  will  be  paid  from  the  House  contingent  fund  rather 
than  committee  budgets. 

Conclusion 

Whether  the  surge  of  interest  in  oversight  will  continue  or  even 
accelerate  is  still  unclear.   For  oversight  is  complex,  difficult,  and 
often  a  thankless  process.   Moreover,  it  may  produce  more  questions  than 
answers.   Legislative  evaluation  of  programs  and  agencies  can  perhaps 
identify  what  isn't  working,  but  it  probably  won't  answer  the  question 
of  "what  works?"  While  the  problems  that  inhibit  legislative  oversight 
still  exist,  there  is  strong  interest  in  this  function,  and  this  could 
mean  that  existing  programs  will  be  made  to  work  better. 
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'PROGRAM  EVALUATION  CONCEPTS",  GENEVIEVE  J.  KNEZO, 
CONGRESSIONAL  RESEARCH  SERVICE 


The  objective  of  this  brief  is  to  discuss  some  of  the  basic  con- 
cepts of  program  evaluation  research,  the  other  part  of  the  theme  of 
this  seminar  series.   These  remarks  are  intended  to  provide  a  frame- 
work to  enhance  the  relevance  of  current  evaluation  research  prac- 
tices to  legislative  oversight  requirements. 

Federally  funded  program  evaluation  studies  cost  about  $130  mil- 
lion in  the  fiscal  year  1974.   This  is  about  a  six-fold  increase  over 
the  cost  of  such  studies  in  the  fiscal  year  1969.   Despite  the  pro- 
liferation of  evaluation  research,  members  of  Congress,  agency  pro- 
gram administrators,  and  social  scientists  have  expressed  consider- 
able dissatisfaction  with  the  quality  and  utility  of  such  research. 
Generally  it  is  acknowledged  that  many  of  the  inadequacies  of  current 
evaluation  practices  come  from  a  lack  of  understanding  between  the 
doers  and  the  users  of  evaluation  about  the  questions  that  evaluation 
can  answer,  and  about  which  techniques  are  most  appropriate  to  answer 
these  questions. 

Two  Types  of  Evaluation  Research 

Federal  agencies  generally  fund  two  types  of  evaluation  studies. 
These  are  process  evaluations  and  impact  evaluations.   They  can  be  dif- 
ferentiated by  their  purposes,  the  techniques  used  to  do  them,  their 
cost,  and  the  time  required  to  do  the  evaluations. 

Process  evaluations  generate  information  to  answer  management 
questions  about  the  operating  efficiency  of  a  program.   For  instance: 
"who  were  the  beneficiaries  or  target  group?  Were  resources  deployed 


71-867  O  -  76  -  5 


56 


efficiently?  What  was  the  dollar  cost  per  beneficiary?  How  many  per- 
sons were  required  to  administer  the  program?  Generally  was  the  pro- 
gram administered  according  to  the  management  or  delivery  principles 
specified  in  the  research  design?" 

Impact  evaluations  answer  questions  about  a  more  intransigent  is- 
sue, that  of  cause  and  effect.   For  instance:   "did  the  program  inter- 
vention strategy  or  treatment  effect  a  change  in  the  participants? 
Was  the  change  the  one  specified  in  the  program?" 

Agencies  use  process  evaluations  to  obtain  information  needed  to 
make  administrative  changes  in  a  program.   The  information  elicited  in 
impact  evaluations  is  used  to  assist  in  making  major  policy  decisions. 
For  instance,  "should  a  social  innovation  or  new  social  program  be  im- 
plemented nationwide?" 

Process  Evaluation 


Process  and  impact  evaluations  require  the  use  of  different  re- 
search techniques.   Process  evaluations  generally  are  non-experimental. 
They  use  data  collected  in  the  normal  course  of  administering  a  pro- 
gram.  Output  data  are  indicated  in  terms  of  input  data,  for  instance 
the  dollar  cost  per  beneficiary.   These  evaluations  are  inexpensive, 
require  little  time  to  do,  can  be  done  after  the  fact,  or  as  a  normal 
monitoring  function.   They  can  be  done  by  program  admins trators,  either 
in  the  agency  or  in  the  field,  or  if  extramurally,  by  public  adminis- 
trators, auditors,  economists,  or  social  scientists.   The  Water  Quality 
Improvement  Act  of  1970,  P.L.  91-224,  is  an  example  of  a  legislatively 
mandated  process  evaluation.   It  requires  the  collection  of  data  on 
the  effectiveness  of  training  programs.   The  act  specifies  as  measures  of 
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effectiveness,  data  on  the  number  of  persons  trained  and  on  occupa- 
tional categories  for  which  training  was  provided.  The  Congress  re- 
quested the  information  in  support  of  making  recommendations  about 
developing  new  training  programs. 

Impact  Evaluation 

Impact  evaluations  typically  use  a  quasi-experimental  or  an  ex- 
perimental research  design.   Since  one  objective  of  the  impact  eval- 
uation is  to  determine  if  a  program  affected  the  participants'  be- 
havior, impact  evaluations  must  control  for,or  factor  out,  other 
events  which  might  have  caused  modifications  anyway  in  the  partici- 
pants' behavior.   This  type  of  research  requires  the  use  of  a  control 
group  and  a  test  group.   The  members  of  each  group  must  be  matched 
on  psychological,  social,  and  economic  characteristics  relevant  to 
the  program.   Since  another  objective  of  the  impact  evaluation  is  to 
determine  whether  the  treatment  should  be  adopted  nationwide,  the 
groups  must  be  selected  randomly.   This  means  that  in  order  for  in- 
ferences or  generalizations  to  be  drawn  about  the  efficacy  of  the 
program  nationwide,  all  persons  who  ultimately  might  be  given  the 
program  should  have  an  equal  chance  of  being  selected  as  members 
of  the  experimental  groups . 

The  program  or  treatment,  such  as  an  income  supplement  or  a  job 
retraining  program  is  given  to  the  test  group.   The  control  group  does 
not  receive  the  program  or  is  given  a  placebo,  that  is  a  different 
type  of  training  program  not  being  evaluated.   Attitudinal  and  be- 
havioral changes  in  both  groups  are  followed  closely.   Other  factors 
which  might  affect  the  program  outcome,  such  as  an  increase  of 
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unemployment  in  the  area,  are  noted  and  considered.   Then  an  estimate 
is  made  of  the  actual  impacts  of  the  program  on  the  test  group. 

Impact  evaluations  are  costly  and  require  a  long  lead-time,  usu- 
ally several  years.   They  use  the  findings  and  techniques  of  social 
science  research,  such  as  hypothesis  testing,  survey  research,  and 
statistical  manipulation  of  the  data  to  draw  and  match  samples,  to 
collect  and  analyze  data  about  the  program's  impact  and  about  other 
trends  or  events  which  might  affect  the  outcome  of  the  program.   Data 
collected  in  the  normal  course  of  administering  the  program  are  not 
sufficient  for  an  impact  evaluation  since  these  evaluations  may  re- 
quire the  collection  of  longitudinal  data  series,  that  is  trend  data 
overtime,  on  hard  to  measure  attitudes  and  behavior.   Appropriate 
data  collection  procedures  usually  have  to  be  built  into  the  program 
before  it  is  implemented. 

Impact  evaluations  may  be  done  on  an  anticipatory  basis  for  pro- 
gram planning  purposes.   An  example  of  a  legislatively  mandated  impact 
evaluation  is  the  Rehabilitation  Act  of  1973,  P.L.  93-112.   The  Act 
requires  the  conduct  of  an  impact  evaluation,  using  test  groups  and 
control  groups,  to  assess  the  impacts  on  participants  of  different 
types  of  rehabilitation  training  programs.   One  objective  is  to  deter- 
mine the  impact  of  the  program  on  the  participants'  ability  to  adjust 
to  disability  and  to  improve  their  job  skills.   The  New  Jersey  Nega- 
tive Income  Tax  Experiment  is  another  example  of  an  experimental  im- 
pact evaluation. 

Reliable  and  valid  information  about  a  program's  impact  can  be 
obtained  only  by  the  use  of  an  experimentally  based  impact  evaluation 
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strategy.  However  there  are  several  drawbacks  to  experimental  impact 
evaluations,  in  addition  to  those  of  cost  and  time  required.   These 
include: 

-  a  lack  of  basic  social  or  behavioral  science  understanding  of 
the  causes  and  effects  of  human  behavior; 

-  problems  of  invasion  of  privacy  due  to  the  requirements  of  data 
collection  procedures  used  in  impact  evaluation; 

-  problems  of  equity  of  Federal  spending  practices  if     Federal 
subsidies  are  given  to  a  test  group  and  not  to  a  control  group  or  to 
the  larger  potentially  recipient  population;  and 

-  problems  which  result  from  diffuse  or  ambiguous  statements  of 
legislative  program  goals  which  prevent  evaluation  researchers  from 
developing  precise  yardsticks  against  which  to  measure  program  ac- 
complishments.  Such  statements  usually  are  vague  because  of  the  need 
for  political  compromise  in  writing  legislation. 

In  should  be  noted  that  quasi-experimental  research  designs,  which 
do  not  always  use  randomly  selected  test  and  control  groups,  can  some- 
times elicit  useful  policy  relevant  impact  information.   However  this 
information  generally  is  not  as  valid  or  reliable  as  that  elicited  from 
experimental  designs.   One  type  of  quasi-experimental  research  design 
is  a  modification  of  the  before  and  after  case  study.   It  compares  pro- 
jected estimates  of  changes  in  participants'  behavior,  where  predictions 
are  based  on  previous  trends  or  social  science  explanations  about  changes 
in  human  behavior,  with  observed  changes  after  program  is  terminated. 
Like  experimental  designs,  quasi-experimental  designs  also  require  pre- 
planning and  a  long  lead  time. 
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Concluding  Observations 

These  concepts  of  program  evaluation  have  several  implications 
for  legislative  oversight.   First,  both  process  and  impact  evalua- 
tions are  useful  for  legislative  oversight.   However  their  purposes 
and  limitations  must  be  understood  clearly.   Process  evaluations  most 
appropriately  are  used  to  generate  information  about  the  administra- 
tive efficiency  of  on-going  programs.   Impact  evaluations  are  used 
for  anticipatory  program  planning  and  to  assess  whether  a  program 
intervention  strategy  should  be  adopted  nationwide.   Second,  when 
reading  evaluation  reports,  legislative  staff  should  be  aware  of 
the  purposes  of  the  evaluation  and  the  techniques  used.   This  same 
caveat  applies  to  the  legislative  staff  person  writing  requirements 
for  evaluation  into  legislation.   Process  evaluations  cannot  gen- 
erate information  about  program  impacts.   The  most  valid  and  re- 
liable impact  evaluation  information  comes  from  evaluations  which 
use  randomly  assigned  control  and  test  groups.   Third,  tradeoffs 
have  to  be  made  about  the  following  factors  when  deciding  whether 
to  use  an  impact  or  a  process  evaluation  strategy:   cost,  lead-time, 
the  need  to  build  requirements  for  data  collection  procedures  into 
authorizing  legislation;  and  the  pros  and  cons  of  developing  pre- 
cise statements  of  legislative  goals. 

In  conclusion,  it  should  also  be  noted  that  evaluation  is  but 
one  of  many  devices  which  can  be  used  to  generate  information  and 
analysis  for  oversight.   Legislative  adoption  of  even  a  pilot  pro- 
gram reflects  a  political  consensus  about  the  desirability  and  feas- 
ibility of  the  program.   No  amount  of  evaluation,  whether  pro  or  con, 
will  alter  that  consensus. 
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"ASK  THE  RIGHT  QUESTIONS,"  DENNIS  L.  LITTLE, 
CONGRESSIONAL  RESEARCH  SERVICE 


In  order  to  conduct  a  meaningful  agency  and  program  evaluation,  we 
need  to  ask  the  right  questions  at  the  right  time,  and  our  inquiry  must 
take  place  at  several  levels.   The  right  questions  can  be  asked  only  if 
the  evaluator  understands  the  agency's  goals,  missions,  program,  program 
objectives,  and  program  and  administrative  evaluation  criteria.   In  most 
cases  such  an  understanding  does  not  exist  and,  consequently,  our  evalu- 
ations are  non-existent  or  of  questionable  utility.   This  analysis  is  in- 
tended to  provide  the  reader  with  a  framework  for  understanding  and  eval- 
uating the  agency's  reasons  for  existence  (goals  and  missions)  and  their 
instruments  (programs)  for  staying  in  existence.   Hopefully,  by  using 
this  framework,  the  reader  can  begin  to  formulate  the  appropriate  evalu- 
ation questions. 

The  goals  of  many  Government  agencies  or  departments  frequently  are 
non-existent,  ill-defined,  historically  irrelevant,  or  hypothetically  im- 
possible.  The  Commissioner  of  the  Social  Security  Administration  has 
been  quoted  as  saying  that  he  is  not  sure  he  knows  what  the  goals  of  the 
Agency  are,  although  he  does  know  what  they  were  in  1935.   If  the  Com- 
missioner, Congressman  Burke  or  the  chief  of  staff  of  the  Ways  and  Means 
Subcommittee  on  Social  Security,  and  the  Congressional  Research  Service's 
top  Social  Security  expert  were  each  to  describe  in  one  half -page  the 
highest  goal  of  the  Agency,  chances  are  good  that  there  would  be  signif- 
icant variations  in  the  three  papers.   The  appropriate  relationships  among 
the  goals,  missions,  and  program  components  are  represented  by  Figure  1. 
As  illustrated  in  Figure  2,  the  highest  goal  of  the  Social  Security 
Administration  might  turn  out  to  be  individual  or  family  economic  security. 
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This  goal  is  composed  of  the  program  missions  of  income  maintenance  and 
reimbursement  for  services  and  commodities.   The  mission  of  income  main- 
tenance must  be  subdivided  further  into  operating  concepts  and  eligibility 
factors,  as  Figure  3  emphasizes. 

These  concepts  are  composed  of  insured  rights,  guaranteed  rights, 
and  needs-based  rights.   Insured  rights  include  retirement,  survivors  and 
disability  insurance  programs.   Guaranteed  rights  programs  are  not  admin- 
istered by  the  Social  Security  Administration.   The  Supplemental  Security 
Income  program  —  adult  categories  of  public  assistance  —  is  an  SSA  needs- 
based  program.   Entitlement  and  post  entitlement  are  elements  which  deter- 
mine eligibility  factors.   These  detailed  components  of  the  income  main- 
tenance mission  are  pictured  in  Figure  4. 

This  model  of  general  structure  and  the  specific  example  of  the  Social 
Security  Administration  provides  a  framework  for  proper  interrogation  at 
the  program  level  by  examining,  at  various  sectors  and  intervals,  the  re- 
lated variables  which  affect  the  perceived  goal.   Prior  to  appropriation 
hearings  the  committee  staff  should  reqvest  from  the  chief  administrative 
officer  of  the  agency  the  following  information: 

1)  What  are  the  five  main  objectives  for  each  program? 

a)  In  an  effort  to  reduce  the  information  overload,  one-line 
statements  should  be  used  to  describe  these  objectives. 

b)  A  list  of  back-up,  alternative  and/or  additional  objectives 
may  be  furnished  separately. 

2)  What  methods  can  be  used  to  measure  progress  toward  these  objec- 
tives? 

a)   One-line  descriptive  statements  of  the  methods  of  measurement 
should  be  formulated. 
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b)  The  selected  methods  should  not  exceed  five  in  number. 

c)  The  estimated  cost  of  obtaining  data  on  each  indicator  on  hand 
if  the  data  are  not  currently  available  and/or  accessible. 

Upon  being  furnished  this  information,  the  committee  should  attempt 
an  evaluation  of  the  objectives  and  indicators  provided.   This  might  be 
done  by  the  committee  staff,  the  Congressional  Research  Service,  the  Gov- 
ernment Accounting  Office,  or  some  combination  of  the  three. 

In  conducting  the  analysis  the  following  questions  should  be  consid- 
ered: 

1)  Do  the  objectives  concur  with  the  stated  goals  of  the  agency? 

2)  Are  these  the  most  important  objectives? 

3)  Do  the  indicators  measure  program  and  administrative  objectives? 

4)  Are  there  better  or  more  meaningful  indicators? 

Upon  completion  of  this  analysis,  the  committee  may  ask  the  agency 
for  additional  information  and  a  detailed  analysis  of  the  indicators. 
The  committee's  questions  to  the  agency  might  include: 

1)  What  is  the  current  level  of  the  indicator? 

2)  What  is  the  desired  level  of  the  indicator? 

3)  Over  the  next  five  years,  what  would  be  the  cost  —  dollars,  man- 
power, space,  etc.  —  to  close  the  gap  between  the  actual  level 
and  the  desired  level  of  the  indicator? 

An  additional  question  not  related  to  the  specific  problem  is:  What 
effect  would  attainment  of  this  objective  have  on  the  attainment  of  other 
agency  Adjectives? 

By  structuring  the  investigator's  perception  of  the  agency,  we  can 
begin  to  structure  our  evaluation  interrogation,  and  thus,  hopefully,  we 
can  ask  the  right  questions  at  the  right  time  at  the  right  level. 


DISCUSSION 

The  issues  raised  after  presentation  of  the  first  four  papers 
dealt  primarily  with  the  meaning  of  the  concept  of  oversight  and 
methods  which  can  be  used  to  determine  priorities  for  legislative 
oversight.   Several  points  were  made.   First,  frequently  oversight 
is  done  by  exception,  meaning  that  many  federally  funded  programs 
operate  efficiently  without  congressional  oversight;  therefore  the 
scarce  resources  available  for  legislative  oversight  should  be  used 
for  handling  only  especially  controversial  or  troublesome  problems. 
Second,  oversight  can  be  considered  a  continuing  process,  whereby 
the  agencies  receive  a  continuous  stream  of  management  reporting 
information,  and  authorizing  and  appropriations  committees  receive 
annual  budgetary  information.   This  may  result  in  information  over- 
load.  It  was  noted  that  perhaps  attempts  should  be  made  to  cut  the 
flow  of  irrelevant  information  to  agencies  and  committees.   Third, 
political  considerations  frequently  enter  into  the  performance  of 
oversight.   However  it  is  important  for  members  of  Congress  to  use 
their  political  barometers  to  identify  important  areas  requiring 
oversight. 
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'CONGRESSIONAL  RESEARCH  SERVICE:   RESOURCES  FOR  OVERSIGHT  AND 

EVALUATION,"  NORMAN  BECKMAN,  DEPUTY  DIRECTOR, 

CONGRESSIONAL  RESEARCH  SERVICE 


The  mission  of  the  Congressional  Research  Service  for  more  than  a  half 
century  has  been  to  provide  the  Congress,  and  only  the  Congress,  with  research 
and  reference  assistance  on  the  complete  spectrum  of  public  policy  issues  that 
Congress  must  consider.   Recently,  the  Service  was  directed  to  establish  as  a 
goal  massive  aid  for  policy  analysis  in  assisting  the  Congress  to  perform  its 
legislative  functions. 

In  moving  toward  this  goal  of  legislative  support,  it  becomes  apparent 
that  there  is  no  really  significant  distinction  between  providing  policy  analysis 
for  pending  legislation  and  providing  analytical  assistance  for  legislative 
oversight  and  evaluation.   These  classically  discrete  legislative  functions  are, 
in  fact,  part  of  a  "push-pull"  continuous  process.   Implementation  of  new 
legislation  creates,  after  a  period  of  time,  the  opportunity  and  need  for  over- 
sight.  The  evaluation  process  inherent  in  oversight  activities  uncovers  the 
need  for  modification  of  existing  legislation,  termination  of  program  authority 
or  enactment  of  entirely  new  legislation. 

This  is  an  important  concept  because  the  superficial  tendency  is  to  think, 
despite  our  checks  and  balances  system,  of  Congress  as  being  concerned  solely 
with  the  law  producing  function  and  the  Executive  with  the  carrying  out  of 
national  policy.   It  is  easy  to  disregard  the  delivery  side  of  policy  before 
the  legislation  is  enacted  and  to  move  on  to  other  matters  once  the  bill  becomes 
a  law.   The  ideal  prescription  for  Congress  is  continuing  review  and  oversight 
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but  the  temptation  and  frequent  practice  are  to  look  for  other  legislative 
opportunities  rather  than  to  monitor  how  a  program  and  policy  is  being  managed. 

The  theory  behind  the  conduct  of  seminars  on  public  policy  issues  organized 
by  the  Congressional  Research  Service  for  Congressional  Members  and  staff  is 
to  collapse  the  time  involved  in  the  transfer  of  ideas.   People  with  expertise 
and  new  knowledge  to  contribute  are  brought  face  to  face  with  people  directly 
involved  in  the  very  dynamic  legislative  process.   The  intervening  and  time 
consuming  publication  process  for  the  transmittal  of  new  information  is  by- 
passed through  informal  presentation  and  back  and  forth  discussion  and  debate. 
In  the  case  of  this  seminar  series  on  Legislative  Oversight  and  Program  Eva- 
luation the  experts  are  "us"  -  the  Congressional  Research  Service,  the  General 
Accounting  Office,  the  Office  of  Technology  Assessment,  and  the  congressional 
staff  speakers. 

The  basic  authority  for  the  Congressional  Research  Service  in  assisting 
the  Congress  to  meet  its  oversight  responsibilities  is  contained  in  the 
Legislative  Reorganization  Act  of  1970.   Several  of  the  Act's  provisions  are 
directed  to  the  Service's  support  of  congressional  oversight  and  evaluation 
activities:   an  explicit  mandate  for  policy  analysis  of  legislative  proposals 
and  oversight,  new  anticipatory  functions-the  submission  of  subject  and  termi- 
nating program  lists  at  the  beginning  of  each  new  Congress  and  a  directive  to 
maintain  continuous  liaison  with  all  committees.   The  Service  has  no  monopoly 
on  this  function-sharing  it  with  a  wide  range  of  resources  within  the  Congress 
(including  Members'  staffs,  professional  staff  of  committees,  policy  committees 
and  informal  research  groups,  the  General  Accounting  Office,  the  Office  of 
Technology  Assessment,  and  the  new  Congressional  Budget  Office),  and  with  such 
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outside  resources  as  the  executive  departments  and  agencies,  the  various  interest 
group  organizations  located  in  Washington  and  elsewhere,  and  universities  and 
private  research  organizations. 

Following  a  very  brief  history  and  introduction  to  the  Service,  this  paper 
speaks  to  four  questions:   What  has  been  done  to  implement  the  specific  new 
statutory  mandates  given  the  Service  in  1970?  What  has  been  done  to  provide 
additional  analytical  support  to  the  Congress  for  oversight?  What  are  some  of 
the  new  directions  being  considered  by  the  Service?  What  are  the  implications 
for  CRS  of  new  proposals  to  improve  Congress'  research  and  information  base? 
Following  this  paper,  three  speakers  from  the  Congressional  Research  Service 
will  present  three  different  case  studies  of  oversight  staff  support  provided 
by  CRS: 

(a)  oversight  activities  stemming  from  CRS  statutory  responsibility 
to  identify  emerging  subject  areas  of  interest  to  committees  and 
report  on  terminating  program  authorities; 

(b)  oversight  opportunities  stemming  from  analysis  of  Executive 
Branch  reporting  to  Congress  in  the  field  of  foreign  affairs;  and 

(c)  direct  support  for  committees'  oversight  activities  in  the  field 
of  health  care. 

Some  other  recently  completed  oversight  projects  conducted  by  CRS  that 
come  readily  to  mind  (in  order  to  preserve  confidentiality  of  the  request  and 
response,  only  general  titles  are  provided)  include:  mineral  leasing;  rede- 
ployment of  U.S.  armed  forces  in  certain  countries;  implementation  of  revenue 
sharing;  community  development;  education  and  recreation  legislation;  and 
retirement  systems  and  immigration. 
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To  emphasize  the  priority  to  be  given  by  the  Service  to  public  policy 
research,  Section  321  of  the  Legislative  Reorganization  Act  of  1970  (Public 
Law  91-510)  changed  the  institution's  name  from  the  Legislative  Reference 
Service  to  the  Congressional  Research  Service.   The  intent  of  this  section 
of  the  act  was  to  expand  and  change  the  nature  of  CRS  support  to  congressional 
committees,  emphasize  the  primary  importance  of  assistance  on  legislative 
matters,-  promote  analytical  and  original  research,  grant  the  Service  greater 
autonomy  within  the  Library  of  Congress  and  render  it  more  directly  responsive 
to  the  Congress. 

What  are  the  resources  available "and  how  is  CRS  organized  to  provide 
oversight  and  evaluation  support?  The  general  intent  of  the  1970  act  was  not 
significantly  different  from  that  of  the  1946  Legislative  Reorganization  Act, 
which  emphasized  committee  support,  appointment  of  senior  specialists,  and 
greater  organizational  independence  for  the  Service.   But  this  idealistic  postwar 
act,  like  the  Employment  Act  of  the  same  year,  never  fully  achieved  its  potential. 
In  the  first  four  years  after  the  1946  act,  appropriations  for  the  Service  in- 
creased from  $213,000  in  1945  to  $514,700  in  1949  and  its  staff  grew  from  62 
persons  to  117  in  1949.   In  contrast,  budget  increases  during  the  first  four 
years  after  the  1970  act  raised  the  Service  from  a  base  of  $5,653,000  and  363 
positions  in  FY  1971  to  $13,345,000  and  703  positions  in  FY  1975. 

The  Congressional  Research  Service  is  organized  into  ten  divisions.   Seven 
of  the  divisions,  staffed  by  senior  specialists,  specialists,  analysts  and 
research  assistants,  perform  substantive  research  in  the  following  broad  subject 
areas:   American  Law,  Economics,  Education  and  Public  Welfare,  Environmental 
Policy,  Foreign  Affairs  (including  national  defense),  Government,  and  Science 
Policy.   The  research  divisions  are  supplemented  by  an  interdisciplinary  group 
of  subject  area  specialists  who  comprise  the  Senior  Specialists  Division,  a 
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Congressional   Reference  Division  which  handles   the  bulk  of  the   ref- 
erence requests,    and   the  Library  Services  Division  which  provides 
information  and  bibliographic  support. 

The  Joint   Committee  on  the  Organization  of   the   Congress,    in  pro- 
posing the  Reorganization  Act   of  1946   had  "endorsed  the  principles 
that    (a)    research   can  be  most  effectively   supplied  by  a  pool   of   in- 
dependent  experts,   and    (b)    that   the  Congress  should  have  direct   ac- 
cess  to  its   own  separate   research  agency."      In  effect,    Congress   re- 
endorsed  those   less-than-fully  achieved  principles   in  the  Legisla- 
tive Reorganization  Act   of  1970,  with  an  added  emphasis   upon  support 
for  the   functions  of   congressional  committees. 

The  House   Committee   on  Rules,    in  its   report   on  the  Reorganiza- 
tion Act  issued  on  June  17,   1970,    emphasized  the  Service's  new  and 
expanded  committee-related  duties.      "The  complexity  of  committee 
responsibilities,"   the   report  said,    "requires   another  supplementary 
staff  to  provide  massive  aid  in  policy  analysis.      For  this   purpose 
we  propose   that    Congress   expand   the   functions   and   facilities   of  the 
Legislative  Reference   Service. . . .      Advance  planning  is   essential  if 
committees   are  to  derive  the  maximum  benefits    from  the   resources    to 
be  made  available  by   the   Congressional  Research   Service." 

A  number  of  organizational   options  were  available   to  the   Con- 
gress  in  achieving  better  oversight   over  the   administration  of  laws 
and  policy  analysis   assistance   in  examining  new  legislative  propo- 
sals.     The  House   Rules   Committee   rejected  making  additional  staff 
available  to  each   committee  on   the   grounds   that   they  would  be   ab- 
sorbed in  day-to-day  work,    could  not   do  specialized,    indepth  work, 
nor  provide   for  exchange  of  information. 
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The  Service  is  now  the  largest  institution  solely  devoted  to  pro 
viding  public  policy  research  and  information  assistance  to  the  Con- 
gress.  It  receives  and  responds  to  over  200,000  requests  a  year, 
which  include  over  400  assignments  for  indepth  analytical  assistance 
requiring  a  significant  amount  of  manpower.    CRS  provides  factual 
information,  analysis  of  issues,  alternatives  to  proposals,  and  eval- 
uation of  alternatives  without  either  advocacy  or  partisan  bias.   It 
also  observes  strict  confidentiality  of  both  requests  and  responses. 
Its  assignments  are  received  from  four  primary  sources — committee 
staffs,  members  and  their  staffs,  constituents  (received  through  con- 
gressional offices) ,  and  specific  statutory  requirements  such  as  pro- 
vision of  lists  of  terminating  programs  and  activities  and  preparation 
of  the  Digest  of  General  Public  Bills  and  Resolutions. 

Implementation  of  New  Statutory  Responsibilities 

The  legislative  history  of  Section  321  of  the  Legislative  Re- 
organization Act  of  1970  indicates  that  the  Congress  sought  to 
achieve  two  principal  goals  in  revising  the  basic  statute  of  the 
Service.   First,  it  was  clearly  intended  that  the  Service's  research 
and  analytical  support  for  the  committees  of  Congress  should  be  ex- 
panded.  Second,  while  sanctioning  CRS  assistance  to  committees  and 
members  in  their  representative  functions  and  activities,  it  intended 
to  make  clear,  as  the  House  Report  put  it,  that  "the  major  function 
of  the  Congressional  Research  Service  is  to  provide  assistance  to 
committees  and  individual  Members  of  the  Congress  with  respect  to 
legislative  matters." 
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The  current  operational  goals  of  the  Service  are: 

-  To  provide  indepth  analytical  and  research  assistance  to  the 
committees  of  the  Congress  to  support  them  in  their  legisla- 
tive and  oversight  responsibilities; 

-  To  provide  both  members  and  committees  with  other  analytical 
and  research  assistance  to  support  their  legislative  respon- 
sibilities; 

-  To  assure  that  CRS  assistance  to  the  Congress  is  of  the  highest 
feasible  quality  and  that  it  is  responsive  to  the  needs  of  the 
Congress ; 

-  To  provide  information  on  the  status  and  content  of  current 
legislative  proposals; 

-  To  provide  both  information  and  reference  assistance  to  both 
members  and  committees  to  support  their  legislative  and  over- 
sight responsibilities; 

-  To  assist  members  of  Congress  with  their  representative  func- 
t  ions ; 

-  To  provide  the  necessary  administrative  support  to  insure  the 
optimal  use  of  CRS  resources  in  meeting  congressional  informa- 
tion and  research  needs. 

Stemming  from  these  goals  are  four  major  new  directives  to  the 
Service  to  be  examined  in  more  detail:   (a)  to  provide  policy  analy- 
sis and  research  to  assist  committees  in  their  legislative  functions; 
(b)  submission  of  lists  of  subjects  and  policy  areas  to  committees 
at  the  opening  of  each  new  Congress;  (c)  submission  of  a  list  of  pro- 
grams and  activities  scheduled  to  terminate  to  each  appropriate  com- 
mittee at  the  beginning  of  each  Congress;  and  (d)  maintenance  of  con- 
tinuous liaison  with  all  committees. 

Policy  Analysis  for  Committees.   The  statute  (P.L.  91-510)  di- 
rects that:   "(a)  Upon  request,  CRS  is  to  advise  and  assist  all  com- 
mittees and  joint  committees  of  Congress  in  the  analysis,  appraisal, 
and  evaluation  of  legislative  proposals  in  order  to  assist  them  (1)  in 
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determining  the  advisability  of  enacting  such  proposals,  and  of  alter- 
natives to  them,  and,  (2)  in  evaluating  alternative  methods  for  accom- 
plishing the  goals  of  such  proposals." 

This  bold  and  idealistic  directive  to  supply  committees  with  ob- 
jective, indepth  analysis  and  presentation  of  alternatives  on  any  sub- 
ject matter  raised  a  number  of  questions  for  CRS.   Is  it  necessary  to 
set  up  a  separate  research  staff  to  provide  these  indepth  services? 
What  is  the  relationship  between  the  jurisdictions  of  the  congres- 
sional committees  involved  and  the  subject  specializations  of  each  of 
the  existing  CRS  divisions?  What  gaps  exist  in  subject  coverage  and 
competence  within  CRS?  What  kinds  of  methodology  would  constitute 
sound  policy  analysis  research  and  what  new  analytical  techniques  and 
resources  should  be  implemented  by  the  Service?  Can  policy  analysis 
leading  to  "determining  the  advisability  of  enacting  each  proposal" 
be  provided  without  violating  CRS  policies  of  nonadvocacy?  How  rapid- 
ly can  the  Service  expect  to  meet  the  large  volume  of  unanticipated 
needs  without  compromising  standards  of  quality  and  timeliness?  How 
are  priority  research  efforts  to  be  determined?  How  can  the  possible 
fears  on  the  part  of  professional  committee  staff  about  the  impact  of 
the  Service's  new  duties  be  allayed?  Should  the  Service  concentrate 
on  committees,  already  receiving  considerable  research  support  or  on 
those  committees  receiving  little  or  no  assistance  at  the  present  time? 
Should  contacts  be  made  at  the  subcommittee  level  or  channeled  through 
the  full  committee?  While  the  Service  has  explored  each  of  these  in 
depth,  most  will  remain  unanswerable  until  further  experience  has  been 
gained. 
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As  part  of  its  evaluation  and  planning  effort,  the  Service  now 
identifies  and  monitors  major  committee  projects  that  require  analy- 
sis, have  a  direct  connection  to  legislation,  and  require  a  signifi- 
cant investment  of  CRS  manpower.   During  FY  1974,  456  such  projects 
were  initiated. 

A  current  analysis  of  CRS  research  for  committees  highlights  a 
number  of  variables  affecting  the  amount  and  type  of  work  CRS  provides 
congressional  committees.   The  analysis  notes  that  even  with  its  new 
mandate,  CRS  progress  is  conditioned  by  factors  which  it  has  little 
or  no  control.   The  basic  decisions  that  determine  both  the  volume 
and  the  type  of  work,  are  made  by  the  committees. 

Lists  of  Subjects  and  Policy  Areas  for  Committees.   One  of  the 
major  new  responsibilities  assigned  to  the  Congressional  Research 
Service  by  the  Legislative  Reorganization  Act  of  1970  was  designed 
to  assist  committees  with  their  advance  planning.   CRS  was  thus  di- 
rected to  make  "available  to  each  committee  of  the  Senate  and  House 
of  Representatives  and  each  joint  committee  of  the  two  Houses,  at 
the  opening  of  a  new  Congress,  a  list  of  subjects  and  policy  areas 
which  the  committee  might  profitably  analyze  in  depth..." 

Here  too  a  range  of  problems  and  options  confronted  the  Service. 
Should  a  comprehensive  list  be  prepared  for  each  committee  or  should 
the  lists  be  limited  to  subjects  on  which  CRS  could  offer  assistance? 
Should  only  lists  be  provided  or  should  additional  analytical  materials 
be  provided?  Should  lists  be  provided  to  committee  chairmen  only,  all 
members  of  a  committee,  or  to  Congress  as  a  whole?  What  methodological 
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procedures  should  be  used  in  identifying  and  tracking  policy  areas  in 
each  major  subject  field?  What  if  a  committee  does  not  desire  the  sub- 
mission of  such  a  list? 

For  the  pilot  implementation  of  this  new  responsibility,  the  fol- 
lowing procedures  were  developed  and  carried  out.   Late  in  19  72,  ap- 
propriate CRS  staff  met  with  committee  staffs  and,  in  some  cases,  mem- 
bers.  Ad  hoc  teams  consisting  of  some  30%  (14A  persons)  of  the  total 
research  staff  were  formed  to  work  with  the  committees.   Teams  were 
composed  of  individuals  knowledgeable  in  most  aspects  of  a  committee's 
jurisdiction  and  in  many  cases  included  staff  from  several  divisions. 
The  committee  was  encouraged  to  discuss  its  plans  and  needs  for  re- 
search with  the  appropriate  team  and  in  turn  CRS  staff  suggested  ad- 
ditional promising  subject  areas  for  the  committee  to  consider.   It 
was  also  an  opportunity  to  describe  CRS  capability  in  the  committee's 
fields  of  interest.   The  resulting  lists  of  subjects  and  policy  areas 
thus  incorporated  contributions  by  both  CRS  and  the  committee. 

Each  committee  was  given  the  option  of  receiving  a  selected  list 
consisting  only  of  subjects  on  which  the  Service  could  provide  assis- 
tance, or  a  comprehensive  list,  which  included  but  was  not  limited  to 
those  subjects  on  the  selected  list.   The  final  product,  unless  spe- 
cifically requested  otherwise  by  a  committee,  included  a  summary  list 
of  agreed-upon  subjects  and  policy  areas,  a  one-page  description  of 
each  subject  to  explain  the  reasons  for  its  inclusion  and  provide  ad- 
ditional sources  of  information,  and,  to  the  extent  feasible  and  de- 
sired by  the  committee,  additional  materials  to  supplement  and  illus- 
trate the  one-page  treatment. 
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Terminating  Programs.   Documentation  and  current  status  of  legis- 
lation has  always  been  a  concern  of  the  Congress.   The  Reorganization 
Act  inaugurated  new  documentation  responsibilities  by  directing  that 
CRS  prepare  for  each  committee  a  list  of  programs  and  activities, 
within  its  jurisdiction,  scheduled  to  terminate  during  each  Congress. 
CRS,  in  developing  its  guidelines,  assumed  that  the  principal  purpose 
of  preparing  the  lists  was  to  further  assist  the  committees  in  their 
advance  planning  efforts. 

Beginning  in  late  19  71,  the  American  Law  Division  of  the  Service 
examined  over  4,000  statutes  enacted  during  the  preceding  decade  and 
by  December  1972  had  identified  some  730  programs  and  activities  sched- 
uled to  terminate  during  the  Ninety-third  and  subsequent  Congresses. 
Over  half  of  these  programs  related  to  authorization  of  appropriations, 
and  an  additional  23%  provided  general  authority  for  programs  operated 
by  executive  branch  agencies.   Other  categories  included  termination 
of  commissions,  loan  authority,  tax  authority,  reservations  of  land, 
pensions,  and  reporting  requirements. 

Following  identification  of  terminating  programs,  the  Service  pre- 
pared summary  lists  of  the  458  programs  and  activities  scheduled  to  ex- 
pire during  the  Ninety-third  Congress  and  transmitted  these,  with  sup- 
plementary descriptive  materials,  to  32  House,  Senate,  and  joint  com- 
mittees of  Congress.   Each  committee  packet  included  basic  identifying 
information  and  a  legislative  history  on  each  terminating  program  with- 
in the  committee's  jurisdiction.   For  319  programs  terminating  during 
the  first  session  of  the  Ninety-third  Congress,  the  Service,  insofar 
as  resources  permitted,  also  sent  to  each  committee  a  more  detailed  re- 
port to  aid  the  committee  in  reviewing  the  program. 
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CRS  Liaison  with  Committees.   The  Legislative  Reorganization  Act 
requires  CRS  to  maintain  continuous  liaison  with  all  committees  so 
that  they  may  better  understand  and  utilize  the  Service's  multidis- 
ciplinary  research  resources  and  facilitate  advance  planning.   Al- 
though separately  stated  in  the  1970  act,  several  of  the  Service's 
new  duties  concerning  congressional  committees  are  closely  related 
in  overall  purpose.   Because  preparation  of  lists  of  subjects  and 
policy  areas  and  terminating  programs  for  committees  was  not  re- 
quired until  late  1972,  the  Service  turned  its  attention  to  the  gen- 
eral liaison  function. 

The  objective  of  the  new  liaison  effort  to  be  undertaken  was  de- 
scribed in  a  memorandum  from  the  director  (contained  in  the  1971  An- 
nual Report)  stating  in  part: 

Our  liaison  efforts  with  committees  should  be  directed  toward: 
(a)  keeping  them  informed  about  our  present  and  potential  re- 
sources for  assisting  them;  (b)  acquainting  them  with  Service 
reports,  already  prepared  or  in  preparation,  of  interest  to 
them;  (c)  informing  them  about  other,  non-Service,  materials 
of  use  to  them  and  of  techniques  by  which  the  Service  can  keep 
them  regularly  informed  about  such  materials;  (d)  the  prepara- 
tion and  submission  of  those  lists  of  subjects  and  policy 
areas  the  committees  might  profitably  analyze  in  depth  that 
we  are  obligated  to  supply  them,  including  subjects  of  emer- 
ging interest  to  them;  and  (e)  preparing  and  submitting  the 
lists  of  expiring  legislation  also  required  by  the  Reorgani- 
zation Act. 

Perhaps  the.  major  options  facing  the  Service  in  implementing  the 
liaison  function  were  those  of  internal  organization  and  rate  of  im- 
plementation.  A  recent  internal  memorandum  states  the  basic  organ- 
izational decision: 

One  of  the  first  questions  the  Service  examined  after  passage  of 
the  1970  Act  was  whether  it  should  establish  a  staff  exclusively 
devoted  to  committee  liaison — that  is  personnel  whose  sole  func- 
tion it  would  be  to  maintain  communication  and  contact  with  com- 
mittees.  After  careful  consideration,  and  at  the  urging  of  vir- 
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tually  every  senior  CRS  staff  with  experience  in  working  closely 
with  committees,  the  Service  decided  not  to  adopt  this  approach, 
at  least  during  the  first  few  years  of  operations  under  the  1970 
Act.  Until  circumstances  indicated  another  course,  therefore, 
we  decided  to  rely  upon  our  senior  subject  people,  those  who  ac- 
tually do  substantive  work  for  committees,  for  the  liaison  func- 
tion. 

While  a  more  formalized  network  might  have  been  expected  to  be 

established  by  this  time,  the  process  to  date  has  been  an  evolving 

one.   This,  in  large  part,  is  due  to  the  number  of  committees  and 

subcommittees  involved,  approximately  350.   The  goal  continues  to  be 

an  evolutionary  approach  designed  to  build  working  relationships  based 

upon  subject  expertise. 

Other  Innovative  Services 

The  Service's  response  to  congressional  needs  goes  beyond  the 
very  ambitious  and  explicit  statutory  provisions  of  the  Legislative 
Reorganization  Act.   The  House  Rules  Committee  report  on  the  Reorgan- 
ization Act  directed  CRS  to  experiment  with  additional  ways  of  pro- 
viding assistance  to  Congress: 

Staffs  should  become  proficient  in  existing  analytical  techniques 
and  develop  new  ones  as  necessary.  Both  agencies  [CRS  and  GAO] 
should  be  encouraged  to  experiment  and  to  be  innovative  in  carry- 
ing out  their  new  as  well  as  their  older  functions. 

The  director  of  the  Congressional  Research  Service,  in  recent 

testimony  before  the  Joint  Committee  on  Congressional  Operations, 

identified  certain  new  services  not  explicitly  required  by  the  1970 

act: 

We  sponsor  several  series  of  seminars  at  which  outside  experts 
discuss  major  public  policy  issues  with  Members  or  staff.   We 
have  steadily  expanded  our  automation  facilities  and  sought 
new  and  promising  applications.   One  of  these,  for  example, 
provides  us,  and  therefore  the  Congress,  with  one  of  the  largest 
and  most  up-to-date  economic  data  bases  currently  available  and 
includes  programs  for  analyzing  the  economic  impact  of  alternative 
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fiscal  and  monetary  policies.   With  the  encouragement  of  the 
Senate  Committee  on  Rules  and  Administration,  CRS  is  develop- 
ing an  issues  briefing  system  through  which  we  will  be  able 
to  provide  concise  updated  information  about  current  legisla- 
tive issues  by  video  screens  located  in  the  offices  of  Members. 

The  public  policy  seminars  for  members  and  staff,  and  the  automated 

information  services,  are  described  in  more  detail  below. 

Public  Policy  Seminars .   The  Service  has  expanded  cooperative 
arrangements  with  outside  organizations  to  permit  members  and  their 
staffs  to  meet  informally  with  experts  on  national  issues.   With 
the  Advanced  Study  Program  of  the  Brookings  Institution,  CRS  ini- 
tiated in  the  fall  of  1972  a  series  of  seminars  for  members  of 
Congress  and  a  separate  series  for  the  senior  staffs  of  member 
and  committee  offices.   In  the  past  two  years,  twelve  seminars 
for  members  have  included  discussion  of  such  topics  as:   U.S.  re- 
lations with  China,  crime  prevention  and  law  enforcement,  energy 
and  the  environment,  national  housing  programs,  consumer  protec- 
tion, media  and  the  government,  and  the  multi-national  corporation. 
Eight  seminars  have  been  held  for  congressional  staff  on:   legis- 
lative implications  of  the  1972  election  results,  budget  reform 
and  control,  legislation  affecting  the  status  of  women,  and  new 
trends  in  strategic  weapons  policy  in  the  national  defense  budget. 
The  series  has  attracted  an  overall  attendance  of  277  members  and 
677  staff. 

Additional  seminar  activities  have  included  a  continuing  series 
on  national  urban  growth  policy,  four  seminars  exploring  basic  tax 
issues,  held  for  congressional  staff  in  1973,  and  presentations  by 
CRS  and  the  Office  of  Management  and  Budget  in  February  1974  to 
House,  Senate,  and  CRS  staff  on  use  of  the  1975  budget  documents. 
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Automated  Information  Services.   The  Service  has  identified  a 
number  of  procedures  with  potential  for  the  storage  and  delivery  of 
information  through  the  application  of  data  processing  techniques. 
Thus,  in  recent  years,  the  service  has  gained  access  to  or  has  in- 
ternally developed  a  number  of  data  files.   The  Legislative  Infor- 
mation Data  Base  includes  the  contents  of  the  Service's  Bill  Digest 
and  additional  information  on  all  public  bills  and  resolutions  of 
the  current  Congress.   Each  bill  is  monitored  for  22  data  elements 
(for  example,  bill  co-sponsor,  committee  action,  identical  bills) 
and  information  is  retrievable  via  video  tube  in  several  formats 
(for  example,  a  run  of  all  bills  assigned  to  a  particular  commit- 
tee and  a  listing  of  all  bills  relating  to  a  specific  subject). 

Most  recently,  at  the  initiative  of  the  chairman  of  the  Senate 
Committee  on  Rules  and  Administration,  the  Service  has  been  devel- 
oping an  automated  Issues  Briefing  System.   This  experimental  new 
service  is  being  made  available  on  a  pilot  basis,  pending  its  fur- 
ther development  and  refinement.   The  system  is  designed  to  serve 
the  current  information  needs  of  members,  committees,  and  their 
staffs  by  maintaining  general,  easily  accessible,  summaries  of  many 
key  issues  of  public  policy.   The  summaries,  or  issue  briefs,  are 
provided  by  means  of  a  computer  information  storage  and  retrieval 
system  that  can  be  accessed  by  video-tube  terminal.   Copies  of  in- 
dividual Issues  Briefs  are  also  available  to  both  the  Senate  and 
House  in  printed  form.   Each  Issue  Brief  (currently  on  130  topics) 
contains  current  information  on  background  facts,  legislation,  con- 
gressional hearings,  reports,  and  chronological  developments,  and 
provides  useful  references. 
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In  addition  to  these  programs  maintained  by  CRS  personnel  and 
stored  in  the  library's  computer,  the  Service  currently  has  rental 
access  to  five  outside  data  banks.   The  New  York  Times  Information 
Bank  is  accessed  by  a  videotube  and  associated  high-speed  printer 
and  queries  are  transmitted  by  direct  telephone  line  to  the  data 
files.   These  files  contain  indexes  and  abstracts  of  all  news  ar- 
ticles published  in  the  New  York  Times  since  1969  and  abstracts 
from  selected  additional  magazines  and  newspapers.   MEDLINE  con- 
tains over  400,000  entries  from  some  1,200  journals  in  the  medical 
research  and  public  health  fields.   Its  bibliographic  data  is  re- 
trievable by  author  or  subject.   JURIS  (Justice  Retrieval  and  In- 
quiry System)  maintains  the  text  of  six  information  files,  in- 
cluding the  U.S.  Code.   Additionally,  CRS  has  access  to  programs 
that  provide  the  program  grants  and  expenditures  of  the  Depart- 
ment of  Health,  Education,  and  Welfare;  that  can  project  the  costs 
of  Federal  support  to  education  and  other  programs;  and  that  pro- 
vide an  econometric  model  of  the  national  economy,  which  enables 
extensive  analytical  review  of  the  Federal  budget. 

The  Past  is  Prologue 

In  the  past  four  years,  CRS  appropriations  requests  have  placed 
the  highest  priority  on  implementation  of  new  and  expanded  responsi- 
bilities under  the  1970  act.   Although  the  rate  of  growth  has  been 
slower  than  anticipated,  the  Service  has  acquired  significant  new 
staff  resources,  and  assistance  to  committees  has  grown  correspond- 
ingly.  Now  that  the  Service  is  moving  toward  completion  of  its  five- 
year  implementation  program,  new  approaches  are  being  explored  to 
satisfy  the  research  needs  of  the  Congress. 
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A  statement  by  the  director  of  the  Service  at  the  May  1974  hear- 
ings of  the  Joint  Committee  on  Congressional  Operations  on  research 
and  information  support  for  the  Congress  sketches  certain  of  these 
exploratory  efforts: 

The  Service  has  already  asked  for  funds  that  will  permit  us  to 
explore  the  addition  of  another  dimension  to  our  support  for 
Congress:   futures  research.   This  relatively  new  discipline 
might  help  us  to  do  a  more  penetrating  job  of  identifying  and 
analyzing  emerging  public  policy  issues.... 

We  also  hope  to  establish  closer  liaison  with  the  scholarly  com- 
munity, perhaps  by  a  program  of  temporary  assignment  of  scholars 
to  the  Service  who  could  provide  tangible  assistance  on  specific 
projects  and  who  would  bring  fresh  insights  to  the  staff  of  CRS 
and  the  Congress. 

Many  Members  and  committees  show  increasing  interest  in  the  use 
of  public  opinion  surveys  to  supplement  their  informational  re- 
sources.  Although  it  is  probably  inappropriate  for  CRS  to  con- 
duct such  surveys,  there  are  a  number  of  ways  in  which  we  might 
assist  Congress  in  this  area. 

I  have  already  mentioned  our  interest  in  establishing  programs 
that  would  permit  us  to  attach  some  of  our  staff  to  committees 
for  limited  periods  so  they  can  learn  about  the  needs  of  those 
organizations . 

The  actions  recently  taken  by  the  House  of  Representatives,  after 
consideration  of  the  recommendations  of  the  Select  Committee  on  Commit- 
tees and  substitute  proposals,  will  also  have  a  number  of  impacts  on 
CRS,  as  well  as  on  the  delivery  of  research  and  information  services 
to  the  Congress  generally.   Of  major  long  term  significance,  there 
was  created  a  study  commission  to  make  recommendations  on  the  infor- 
mation problems  of  the  House  of  Representatives.   The  study  is  to  in- 
clude, but  not  be  limited  to,  CRS,  the  General  Accounting  Office,  and 
the  Office  of  Technology  Assessment  and  the  organizational  framework 
that  makes  them  effective  or  ineffective.   The  House  also  created  a 
legislative  classification  office  and  required  the  preparation  of  sum- 
nary  descriptions  of  House  bills. 
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The  enactment  of  the  Congressional  Budget  Act  of  1974  (P.L.  93-344) 
will  also  significantly  affect  the  work  of  the  CRS.   As  previously  noted, 
the  Service  has  already  strengthened  its  capacity  to  perform  certain 
kinds  of  budget  analyses.   It  is  anticipated  that  cooperative  working 
relationships  will  be  developed  with  the  new  Congressional  Budget  Of- 
fice, as  directed  in  the  statute,  and  with  the  budget  committees. 

Other  legislation  introduced  in  the  Ninety-third  Congress  to  es- 
tablish new  offices  and  commissions  in  the  legislative  branch  would, 
if  enacted,  also  impact  on  CRS  and  its  plans  for  further  implementation 
of  the  1970  Legislative  Reorganization  Act. 

Implicit  in  certain  of  the  proposals  for  improved  congressional 
decision-making  is  a  search  for  a  "philosopher's  stone,"  a  device 
that  will  produce  the  right  answers:   a  deus  ex  machina  that  will 
recommend  legislation  that  best  achieves  the  public  interest. 

The  Service,  as  a  staff  arm  of  a  policy-mediated  and  partisan 
power  structure,  operates  on  an  entirely  different  premise.   It  has 
no  official  views  regarding  which  is  a  better  solution  to  a  legisla- 
tive problem.   It  attempts  to  identify  choices  and  to  forecast  the 
apparent  implications  of  the  alternatives,  but  it  is  the  member  and 
then  the  Congress  that  make  the  decisions. 

Hairline  distinctions  must  be  observed  by  each  CRS  staff  person 
on  the  one  hand  to  avoid  either  advocacy  or  sterility  and  on  the  oth- 
er to  maintain  a  reputation  as  a  leading  specialist  in  his  subject 
field.   The  rules  of  the  game  are  described  in  a  recent  Praeger  vol- 
ume on  U.S.  government  agencies,  The  Library  of  Co'ngress : 

In  order  for  the  CRS  to  survive,  the  congressional  inquirer  must 
never  be  able  to  guess  where  the  researcher  himself  stands  on  any 
particular  issue.   The  result  is  that  if  someone  believes  deeply 
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that  a  particular  solution  is  the  only  answer,  that  person  is 
inappropriate  for  the  CRS....  But  the  person  who  believes  deep- 
ly in  the  importance  of  his  subject  field  and  who  is  eager  for 
the  Congress  to  have  as  many  solutions  available  to  it  as  pos- 
sible and  to  have  as  great  a  knowledge  of  the  impact  of  these 
alternatives,  together  with  a  sensitivity  to  what  these  solu- 
tions will  cost  this  generation  and  five  generations  to  come, 
and  who  yet  recognizes  that  for  the  democratic  system  to  sur- 
vive the  choice  of  a  solution  must  be  left  to  the  elected  rep- 
resentatives— that  is  the  person  the  CRS  seeks  for  its  staff. 

These  and  other  proposals  will  continue  to  involve  CRS  and  the 
Congress  in  carrying  out  the  spirit  of  the  Legislative  Reorganization 
Act  of  1970.   The  CRS  mission  remains  to  respond  to  the  research,  ana- 
lytical, and  informational  demands  of  Congress.   To  do  so  will  mean 
constantly  seeking  and  implementing  new  and  improved  ways  of  fulfil- 
ling that  mission  within  the  ground  rules  of  confidentiality,  non- 
advocacy,  and  nonpartisanship. 
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"THE  SERVICE'S  RESPONSIBILITIES  FOR  EMERGING  ISSUES  AND  TERMINATING  PROGRAMS' 
DAVID  E.  GUSHEE.  CONGRESSIONAL  RESEARCH  SERVICE 


The  Legislative  Reorganization  Act  of  1970  charged  CRS  with  two 
specific  functions  that  relate  at  least  partially  to  congressional 
oversight  activities.   These  two  CRS  functions  are: 

(1)  To  prepare  and  present  to  each  committee  at  the  beginning 
of  each  Congress  a  list  of  subjects  and  policy  areas  the  committee 
might  profitably  pursue  in  depth. 

(2)  To  make  available  to  each  committee  at  the  beginning  of  each 
Congress  a  list  of  programs  and  activities  scheduled  to  expire  during 
that  Congress. 

Since  enactment  of  the  Legislative  Reorganization  Act,  CRS  has 
carried  out  these  new  statutory  functions  twice.   What  follows  is  a 
description  of  how  the  functions  were  carried  out,  what  the  list  so 
generated  contained,  how  the  committees  have  responded,  and  what  the 
exercises  might  mean  to  the  oversight  and  program  evaluation  responsi- 
bilities of  the  committees. 

Basically,  the  CRS  organizes  around  issues  areas,  some  of  which  are 
discipline-based  such  as  law  or  economics,  while  others  are  function- 
based,  such  as  education,  taxation,  environment,  or  governmental  structure. 
Either  way,  CRS  analysts  end  up  specializing  in  reasonably  well  defined 
issue  areas  which  may  or  may  not  coincide  with  committee  jurisdictions 
in  the  Congress. 

Some  months  prior  to  the  end  of  a  Congress,  the  Service  sets  up  ad 
hoc  task  forces  corresponding  roughly  with  committee  jurisdictions.   These 
task  forces  review  the  information  available  to  them  about  issues  facing 
their  committees  in  the  forthcoming  Congress,  and  develop  a  list  of  these, 
along  with  a  brief  analysis  of  the  status  of  debate  on  each. 
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Input  to  these  analyses  includes  a  list  of  programs  within  that 
committee's  jurisdiction  which  are  scheduled  to  expire  (how  this  is 
generated  will  be  discussed  later),  the  analyst's  knowledge  of  the  com- 
mittee's agenda,  an  awareness  of  debate  in  the  current  Congress  on  ini- 
tiatives not  yet  enacted  into  programs,  an  awareness  of  debate  over 
agency  performances  in  implementing  previously  enacted  legislation, 
awareness  of  discussion  in  the  professional  and  trade  literature  or 
"invisible  college"  in  the  issue  areas  or  related  areas  and  awareness 
of  concerns  and  suggestions  for  improvement  of  the  design  of  legislation, 
among  other  things. 

The  ad  hoc  task  forces  thrash  out,  from  this  raw  material,  their 
view  of  issue  areas  their  committees  might  either  have  to  or  want  to 
deal  with,  along  with  some  preliminary  views  on  priorities  and  opportunities 
for  useful  research  and  analysis  the  Service  might  perform  (or  that  others 
might  perform,  if  appropriate). 

The  task  force  coordinators,  thus  armed,  initiate  discussions  with 
committee  staff  beginning  at  the  staff  director  level  and  proceeding  as 
far  into  the  committee  and  subcommittee  staffs  as  can  be  arranged.   These 
have  at  least  two  purposes: 

(1)  To  reach  agreement  on  a  list  of  subjects  and  policy  areas  for 
formal  transmittal  from  the  Service  to  the  committee. 

(2)  To  stimulate  advance  planning  in  the  committees  and  to  include 
in  such  planning  such  analytical  and  other  support  services  the  CRS  might 
constructively  contribute  as  extensions  to  the  committee  staff's  capabilities 
of  time,  resources,  special  expertise,  or  objectivity. 
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In  some  of  these  discussions,  other  purposes  are  served  beyond  these. 
CRS  subject  specialists  might  deal  directly  with  subcommittee  or  other 
staffers  dealing  with  various  issue  areas,  thus  establishing  a  rapport 
from  which  future  CRS  support  activities  develop.   In  other  cases,  specific 
research  projects  are  developed  and  agreed  upon  on  the  spot,  thus  giving 
the  committee  a  planned  flow  of  reports  and  giving  the  Service  a  chance 
to  plan  its  own  research  agenda  and  resource  development. 

In  other  cases,  of  course,  the  exercise  results  primarily  in  a 
formal  transmittal  only.   Further  work  relationships  are  left  for  develop- 
ment by  other  methods. 

For  the  94th  Congress,  the  Service  organized  task  forces  for  49 
committees.   The  lists  when  transmitted  all  included  some  oversight  topics, 
and  a  number  of  them  included  agreements  on  studies  the  CRS  would  either 
do  or  participate  in  as  components  of  the  committee's  oversight  activities. 

Subsequent  to  transmittal  of  the  lists,  of  course,  contacts  between 
committees  staffers  and  CRS  specialists  have  led  to  many  additional  over- 
sight-related analyses  by  CRS  for  committees.   We  don't  know  for  sure 
how  many  of  these  are  direct  or  indirect  results  of  the  subject  list 
exercise. 

I  mentioned  at  the  beginning  that  the  CRS  ad  hoc  task  forces  used 
as  input  to  their  own  discussion  the  lists  of  programs  scheduled  to 
terminate  during  the  next  Congress.   These  lists  are  the  result  of  a 
program  initiated  after  the  Reorganization  Act  of  1970,  whereby  each 
new  Public  Law  is  scanned  to  identify  provisions  with  specified  termi- 
nation dates. 

Each  such  program  is  entered  on  a  computer  file  with  a  brief  descrip- 
tion, a  statutory  citation,  the  expiration  date,  the  committees  of  juris- 
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diction,  and  a  brief  legislative  history.   These  entries  are  sorted  by 
expiration  date  and  by  committee. 

For  the  93rd  Congress,  there  were  458  terminating  programs,  319  of 
which  were  scheduled  to  expire  in  1973.   About  55%  of  the  458  were  authori- 
zations of  appropriations;  the  next  largest  categories  were  reports  from 
agencies  and  commissions  and  "general  authority." 

For  the  94th  Congress,  421  programs  have  been  identified  as  expiring. 
The  breakdown  among  categories  is  similar  to  that  for  the  93rd  Congress. 
Most  lists  transmitted  at  the  beginning  of  this  Congress  included  issues 
associated  with  these  terminating  programs,  and  most  terminating  appropri- 
ations authorizations  had  issues  associated  with  them  that  were  controver- 
sial enough  to  be  included,  either  explicitly  or  implicitly,  in  the  issues 
treated. 
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"MONITORING  REPORTS  TO  CONGRESS  REQUIRED  BY  LAW", 
ELLEN  C.  COLLIER,  CONGRESSIONAL  RESEARCH  SERVICE 


The  requiring  of  a  report  from  the  executive  branch  is  one  of  the 
oldest  and  most  common  methods  by  which  Congress  seeks  to  oversee  the 
carrying  out  of  the  legislation  it  enacts.  When  a  law  contains  a  pro- 
vision such  as  that  "The  President  shall  report  to  Congress  by  January 
30  of  each  year  on  actions  taken  under  this  Act,"  Congress  demonstrates 
its  intention  to  follow  through  and  see  how  the  legislation  is  carried 
out.   That  it  is  an  old  and  common  method  is  shown  by  the  fact  that  since 
1821  the  rules  of  the  House  of  Representatives  have  required  that  each 
year  the  Clerk  present  to  each  member  a  list  of  all  reports  which  each  ex- 
ecutive agency  is  required  to»  submit.   The  list  submitted  on  January  14, 
1975  (House  Document  94-7),  was  44  pages  long.   But  perhaps  because  re- 
porting requirements  are  so  frequent,  they  are  often  not  appreciated  as 
the  powerful  tool  for  obtaining  information  that  they  are. 

There  are  many  different  kinds  of  reporting  requirements.   The  most 
usual  are  for  periodic  reports,  particularly  an  annual  report.   However, 
sometimes  when  Congress  wants  to  keep  a  little  closer  watch  on  a  program 
or  agency,  it  calls  for  a  semi  annual  or  quarterly  report.   Another  kind 
is  for  a  non-recurring  report  as  when  an  agency  is  called  upon  by  Con- 
gress to  make  a  single  study  and  report  its  findings  by  a  specific  date. 
Still  another  kind  is  the  requirement  that  Congress  be  notified  of  cer- 
tain Presidential  determinations  or  findings,  or  of  executive  actions. 
For  example,  it  is  required  that  all  international  agreements  not  sub- 
mitted as  treaties  be  transmitted  to  Congress  within  60  days. 
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Reporting  Requirements  Study 

The  effort  to  monitor  reports  required  in  the  foreign  affairs  field 
began  in  1969  when  the  Senate  Foreign  Relations  Committee  asked  the  For- 
eign Affairs  Division  for  an  identification  and  analysis  of  the  report- 
ing requirements  in  legislation  on  foreign  relations.   The  subsequent 
study,  "Reporting  Requirements  in  Legislation  on  Foreign  Relations," 
published  in  1970,  described  a  patchwork  of  more  than  a  hundred  report- 
ing requirements  spread  through  approximately  50  pieces  of  legislation. 

In  a  few  instances,  such  as  the  Foreign  Assistance  Act,  there  were 
large  numbers  of  reporting  requirements,  some  of  them  overlapping  or 
clearly  obsolete.   In  other  cases,  there  were  no  reporting  requirements 
at  all.   For  example,  when  Congress  established  the  Central  Intelligence 
Agency,  it  did  not  include  any  requirements  for  CIA  to  report  to  Congress. 
Nor  were  there  any  reports  required  from  the  Department  of  State  on  its 
operation  or  on  overall  U.S.  foreign  policy. 

Even  in  the  first  study  it  became  clear  that  the  reporting  require- 
ments and  the  reports  filed  in  response  to  them  could  be  a  valuable  means 
of  communication  between  the  legislative  and  executive  branches.   For  Con- 
gress, they  could  be  an  important  tool  for  overseeing  the  carrying  out 
of  legislation.   However,  whether  they  were  actually  serving  as  such  a 
tool  required  further  study.   There  were  two  major  questions,  namely,  were 
the  reporting  requirements  being  met,  and  was  full  use  being  made  by  Con- 
gress of  the  reports  when  they  were  submitted. 
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Study  on  Required  Reports 

The  Senate  Foreign  Relations  and  House  Foreign  Affairs  Committees 
then  joined  together  and  asked  the  Foreign  Affairs  Division  to  make  a 
survey  of  the  extent  to  which  the  reporting  requirements  were  being  ful- 
filled by  the  executive  branch.  For  the  next  two  years,  1971  and  1972, 
the  Division  kept  records  of  the  date  each  report  was  submitted  in  re- 
sponse to  a  reporting  requirement,  and  examined  every  such  report  to  see 
if  it  fully  met  the  reporting  requirement  and  was  generally  useful  for 
congressional  purposes. 

The  basic  findings  were  published  as  a  joint  committee  print  in 
April  1973  in  the  study  "Required  Reports  to  Congress  in  the  Foreign  Af- 
fairs Field."  The  study  found  that  the  reporting  requirements  in  legis- 
lation and  the  reports  submitted  in  response  to  them  constituted  one  of 
the  major  information  systems  used  by  Congress.   A  substantial  amount  of 
information  was  being  furnished  through  the  ever-growing  number  of  re- 
ports.  Nevertheless,  despite  the  large  number  of  reporting  requirements 
and  reports,  the  reporting  requirement  system  was  not  being  used  to  its 
fullest  potential.   It  seemed  clear  that  the  effectiveness  and  efficiency 
of  the  reporting  requirement  system  could  be  greatly  enhanced. 

A  number  of  common  problems  were  found  in  the  reports  examined.   The 
most  obvious  was  lateness.   A  large  number  of  reports  were  submitted  too 
long  after  the -period  they  covered  to  be  considered  current,  or  too  late 
to  be  useful  in  the  hearings  on  a  program.   Many  reports  had  security 
classifications  and  made  no  attempt  to  present  a  separate  unclassified 
version,  making  it  extremely  difficult  for  any  of  the  information  in  the 


reports  to  be  utilized.   Even  some  of  the  unclassified  reports  were  dif- 
ficult to  obtain.   A  surprisingly  large  number  of  reports  were  not  dated, 
and  many  were  not  clearly  identified  as  to  which  reporting  requirement 
they  fulfilled. 

On  more  fundamental  matters,  while  some  reports  were  excellent,  many 
simply  did  not  contain  enough  information  to  be  useful  to  Congress.   Often 
there  appeared  to  be  a  real  lack  of  understanding  of  the  kind  of  informa- 
tion that  Congress  needed. 

The  study  found  that  required  reports  served  a  variety  of  functions, 
and  that  sometimes  this  led  to  a  deviation  from  the  main  objective.   For 
example,  one  function  of  requiring  a  report  is  often  to  assure  the  avail- 
ability of  information  to  the  public  on  a  program.   Sometimes,  however, 
agencies  submitted  reports  which  probably  were  useful  to  those  agencies 
for  public  relations  purposes  but  which  were  of  very  little  use  to  Con- 
gress in  assessing  the  value  and  needs  of  the  program.   For  example,  some 
annual  reports  were  attractive  brochures  but  not  the  serious  discussions 
of  funds  and  problems  needed  by  Congress.   The  problem  did  not  always  lie 
in  the  report.   Often  the  reporting  requirements  themselves  were  vague 
and  did  not  indicate  when  a  report  was  wanted  or  what  information  was  in- 
tended. 

The  study  indicated  that  although  the  executive  branch  prepared  the 
reports  and  could  do  a  lot  on  its  own  to  improve  them,  there  was  also 
much  that  Congress  needed  to  do.   First,  it  needed  to  develop  a  system  of 
monitoring  the  receipt  of  reports,  so  that  there  would  be  a  central  place 
for  knowing  whether  a  report  had  been  received,  and  to  whom  it  had  been 
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sent.   With  such  a  monitoring  system,  if  a  required  report  was  not  submit- 
ted in  a  timely  fashion,  a  question  could  be  raised  by  the  appropriate 
committee  to  the  agency.   Committees  needed  to  gear  the  use  of  the  reports 
into  their  preparations  for  hearings,  so  that  the  presentation  material 
and  questions  at  the  hearings  could  build  on  the  basic  information  in  the 
reports.   Evidence  that  the  reports  submitted  were  being  used  was  needed 
to  convince  the  preparers  of  the  reports  that  their  efforts  were  worth- 
while. 

Follow-up  Study 

After  the  study  on  required  reports  was  completed  in  April  1973,  the 
chairmen  of  the  House  Foreign  Affairs  and  Senate  Foreign  Relations  Com- 
mittees sent  it  to  the  principal  foreign  affairs  agencies  and  asked  them 
for  comments  on  the  individual  analyses  of  their  reports.   In  addition, 
they  asked  the  Foreign  Affairs  Division  to  continue  to  monitor  the  re- 
ports in  the  foreign  affairs  field  and  to  assist  in  efforts  to  improve 
the  reporting  requirements. 

A  joint  committee  print  containing  the  replies  of  the  agencies  and 
discussing  the  improvement  effort  was  then  published  in  April  1974  under 
the  title  "Improving  the  Reporting  Requirement  System  in  the  Foreign  Af- 
fairs Field."  That  study  pointed»out  that  1973  had  been  a  landmark  year 
in  the  use  of  reporting  requirements,  inasmuch  as  the  potential  of  such 
requirements  had  been  applied  to  the  fullest  in  the  War  Powers  Resolution 
(P.L.  93-148),  passed  over  the  veto  of  the  President  on  November  7,  1973, 
to  place  a  check  on  the  President's  use  of  troops  in  hostilities  in  the 
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absence  of  a  declaration  of  war.   While  the  final  sanction  was  a  require- 
ment for  congressional  action  if  such  hostilities  were  continued  more 
than  60  days,  reporting  requirements  were  made  the  backbone  of  the  plan 
to  assure  consultation  with  Congress. 

The  study  also  showed  two  other  needs.   First,  a  long-range  and  con- 
tinuing effort  was  needed  if  reports  were  to  be  improved.   Even  when  all 
agree  on  the  objective,  it  takes  time  to  define  the  information  needed 
by  Congress,  additional  time  to  change  reports  to  meet  new  specifications, 
and  still  additional  time  after  a  report  is  issued  to  assess  its  useful- 
ness in  the  new  form.   Second,  the  reporting  requirement  system  was  an 
area  in  which  cooperation  between  the  executive  and  legislative  branches 
could  result  in  better  serving  the  needs  of  both. 

Continued  Monitoring 

The  Foreign  Affairs  Division  is  now  assisting  the  Senate  Foreign 
Relations  and  House  International  Relations  Committees  in  monitoring  re- 
quired reports  on  a  continuing  basis.   The  basic  tool  with  which  the 
Foreign  Affairs  Division  monitors  reporting  requirements  is  a  chart  con- 
taining a  list  of  all  the  reporting  requirements  which  are  contained  in 
legislation  under  the  jurisdiction  of  the  two  committees  or  other  legis- 
lation calling  for  reports  which  would  provide  information  relating  to 
foreign  policy.   The  chart  has  six  columns  for  listing  (1)  the  legal  ci- 
tation, (2)  a  brief  digest  of  the  requirement,  (3)  the  agency  by  which 
the  report  is  prepared,  (4)  the  committee  or  receiver  in  Congress, 
(5)  the  date  required  and  (6)  the  date  any  report  is  received  under  the 
requirement.   A  sample  page  of  this  chart  is  attached. 
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From  the  basic  chart,  which  has  some  300  requirements,  a  wall  chart 
has  been  devised  for  keeping  a  record  of  approximately  60  requirements 
in  legislation  handled  by  the  two  committees  which  call  for  reports  on  a 
regular  basis  or  by  a  specific  date.   This  wall  chart  makes  readily  ap- 
parent whether  reports  are  available  or  due,  or  whether  they  are  late. 

In  conclusion,  there  are  two  features  of  this  project  which  are 
worth  noting.   First,  the  work  has  been  done  at  the  joint  request  of  the 
Senate  Foreign  Relations  and  House  International  Relations  Committees. 
This  has  permitted  an  economical  use  of  the  Foreign  Affairs  Division  man- 
power, by  serving  two  committees  at  once,  as  well  as  providing  for  a 
three-way  exchange  of  ideas  and  information  among  the  House  committee, 
the  Senate  committee,  and  the  Congressional  Research  Service. 

Second,  the  work  has  been  sustained  over  a  long  period.   The  com- 
mittees have  followed  through  on  findings  and  recommendations  and  dem- 
onstrated their  continued  interest  in  required  reports  instead  of  drop- 
ping the  subject  after  a  brief  exploration. 

The  Foreign  Affairs  Division  study  has  dealt  only  with  reports  in 
the  foreign  affairs  field,  where  information  is  a  particular  problem 
for  Congress.   However,  many  of  its  findings  would  apply  to  reports  in 
all  fields.   The  oil  crisis  illustrated  that  foreign  affairs  is  not  the 
only  field  where  lack  of  information  is  a  serious  problem.  A  study  for 
the  Commission  of  Government  Procurement  found  very  much  the  same  situ- 
ation in  regard  to  reporting  requirements  in  the  procurement  field  that 
existed  in  the  foreign  affairs  field. 
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On  every  subject  the  requiring  of  report  offers  a  way  to  generate 
and  obtain  needed  information.   Since  the  legislative  requirement  is  the 
first  element  in  this  information  system,  it  should  be  clear  and  well 
thought  out.   The  second  element  is  the  report  filed  in  response  by  the 
executive  branch.   Since  it  is  the  heart  of  this  information  system,  it 
should  be  timely  and  fully  responsive  to  the  needs  of  Congress.   The 
final  element  is  the  use  of  the  report  made  by  Congress.   Since  this  is 
what  determines  whether  the  whole  information  system  is  worthwhile,  it 
is  incumbent  on  Congress  to  develop  procedures  to  assure  that  the  re- 
ports are  fully  used  for  the  oversight  and  evaluation  for  which  they 
were  intended,  or  that  the  requirements  for  them  are  modified  or  deleted. 
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DISCUSSION 

One  major  point  was  raised  in  the  discussion  following  the  pre- 
ceding three  papers.   It  was  noted  that  generally  the  staff  of  the 
Congressional  Research  Service  use  their  own  independent  resources 
and  experienced  judgments  when  developing  emerging  issues  subject 
lists.   However,  it  was  acknowledged  that  some  lists  may  represent 
a  compromise  and  consensus  of  the  issues  raised  by  CRS  suaff  and 
staff  of  respective  committees  who  will  use  the  lists. 
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["THE  GENERAL  ACCOUNTING  OFFICE'S  RESPONSIBILITIES  FOR  OVERSIGHT  AND  EVALUATION 
IN  REFERENCE  TO  THE  CONGRESSIONAL  BUDGET  AND  IMPOUNDMENT  CONTROL  ACT"  ] , 
JAMES  P.  OLIVER,  GENERAL  ACCOUNTING  OFFICE 


There  are  four  phases  in  the  Congressional  Budget  Process:   The 
first  begins  when  the  President  submits  the  current  services  budget  by 
November  10.   It  will  provide  a  budget  based  on  a  continuation  of  ex- 
isting programs  at  the  same  level  of  effort  and  will  take  into  account 
economic  factors.   It  is  to  provide  a  basis  for  the  Joint  Economic 
Committee's  review  of  the  current  services  budget  and  submission  of  the 
economic  evaluation  of  the  budget.   It  also  provides  a  basis  for  the 
standing  committees  to  begin  analyzing  the  programs  for  which  they  are 
responsible. 

Next,  the  President  submits  the  U.S.  Budget  for  the  ensuing  fiscal 
year  15  days  after  Congress  reconvenes.   After  the  President's  budget 
submission,  all  standing  committees  have  a  new  responsibility.   They 
have  to  submit  their  views  on  the  budget  to  the  appropriate  budget  com- 
mittee.  All  committees  submit  their  views  on  the  budget  to  budget  com- 
mittees by  March  15. 
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Committee  Reports  For  The  Budget  Committees 

-  The  act  requires  that  reports  concerning  the  budget  be  prepared 
by  every  standing  committee  of  both  houses  by  March  15. 

-  These  reports  are  to  be  submitted  to  the  committees  on  the  budget 
of  both  houses  and  are  to  include  each  committee's  views  and 
estimates  with  respect  to  the  content  of  the  proposed  concurrent 
resolution  on  the  budget  as  it  relates  to  matters  within  their 
jurisdiction. 

-  Estimates  are  to  be  submitted  of  the  total  amounts  of  new  budget 
authority,  and  budget  outlays  resulting  thereform,  to  be  provided 
or  authorized  in  all  bills  and  resolutions  within  the  jurisdiction 
of  the  committee  for  the  ensuing  fiscal  year. 

The  report  is  particularly  important  for  the  authorizing  committees 
since  it  represents  their  opportunity  to  justify  the  value  of  their  pro- 
grams.  We  anticipate  that  they  will  want  program  evaluation  information 
for  this  purpose. 

The  Congressional  Office  submits  a  report  to  the  budget  committees 
by  April  1.   The  report  includes  5  year  projections  and  a  discussion  of 
national  priorities  and  alternatives. 

After  getting  the  input  from  committees  and  the  Congressional  Budget 
[Office,  each  budget  committee  submits  its  proposed  concurrent  resolution 
by  April  15.  The  scope  of  the  concurrent  resolution  is  important  —  Notice 
point  (2)  —  This,  in  effect,  is  Congress's  view  on  national  priorities. 

First  Concurrent  Resolution 

(1)  The  appropriate  level  of  total  budget  outlays  and  of  total  new  budget 
authority; 

(2)  An  estimate  of  budget  outlays  and  an  appropriate  level  of  new  budget 
authority  for  each  major  functional  category,  for  contingencies,  and 
for  undistributed  intragovernmental  transactions,  based  on  allocations 
of  the  appropriate  level  of  total  budget  outlays  and  of  total  new 
budget  authority; 
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(3)  The  amount,  if  any,  of  the  surplus  or  the  deficit  in  the  budget  which 
is  appropriate  in  light  of  economic  conditions  and  all  other  relevant 
factors; 

Notice  point  (5)  —  This  has  been  almost  exclusively  an  executive 

initiative  in  the  past. 

For  Congress  to  develop  a  meaningful  concurrent  resolution,  it  is 
going  to  need  good  information,  including  program  evaluations,  and  it 
will  need  it  early  in  the  session. 

The  April  15  submission  of  the  first  resolution  completes  Phase  1 
of  the  budget  process.  Phase  2  starts  with  debate  in  both  Houses  under 
stringent  time  limitation  leading  to  — 

First  Concurrent  Resolution  Continued 


(4)  The  recommended  level  of  federal  revenues  and  the  amount,  if  any,  by 
which  the  aggregate  level  of  federal  revenues  should  be  increased  or 
decreased  by  bills  and  resolutions  to  be  reported  by  the  appropriate 
committees; 

(5)  The  appropriate  level  of  the  public  debt,  and  the  amount,  if  any,  by 
which  the  statutory  limit  on  the  public  debt  should  be  increased  or 
decreased  by  bills  and  resolutions  to  be  reported  by  the  appropriate 
committees;  and 

(6)  Such  other  matters  relating  to  the  budget  as  may  be  appropriate  to 
carry  out  the  purposes  of  this  act. 

Completion  of  congressional  action  on  the  first  concurrent  resolu- 
tion by  May  15. 

Also  required  by  May  15  is  the  reporting  of  all  authorizing  legis- 
lation. 

In  future  years  it  will  be  easier  for  the  authorizing  committees 
to  make  this  deadline  since  the  act  requires  the  executive  to  make  ad- 
vance requests  for  authorizing  legislation  on  or  before  May  15.   For 
example,  on  May  15,  1975,  the  executive  branch  must  submit  its  fiscal 
year  1977  authorization  requests. 
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Phase  3  of  the  budget  process  begins  May  16.  It  consists  of  review 
and  enactment  of  spending  bills. 

There  is  also  additional  input  from  the  executive  branch  at  two 
points  in  time.   The  President  may  transmit  an  amended  budget  on  or  before 
April  10  and  July  15. 

Congress's  review  and  enactment  process  continues  through  the  summer 
until  it  completes  action  on  new  budget  authority.   This  is  required  by 
the  seventh  day  after  Labor  Day. 

Then,  Congress  completes  action  on  the  second  concurrent  resolution 
by  September  15,  directing  committees  concerned  to  take  required  action. 
Phase  A  between  Sept.  16  and  Sept.  25  consists  of  taking  whatever  action 
is  required  to  complete  action  on  a  reconciliation  bill  by  Sept.  25. 

The  budget  process  is  to  be  completed  5  days  before  the  new  fiscal 
year  begins. 
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The  Congressional  Information  Services  Group  in  the  Financial  and 
General  Management  Studies  Division  has  been  organized  to  assist  in  pro- 
viding this  information.   The  group  has  established  five  missions  to  be 
performed  — 

For  This  Process  To  Work  And  Produce  Better  Budgets  In  Support  Of  National 
Needs,  Information  Must  Be  Provided  To: 

.  Congressional  Budget  Committees 
.  Appropriations  Committees 
.  Other  Committees 
.  Congressional  Budget  Office 
.  General  Accounting  Office 

There  are  staff  activities  addressing  these  missions.   Some  activi- 
ties participate  in  all  missions;  others  do  not  as  indicated. 
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DISCUSSION 

A  member  of  the  audience  asked  Mr.  Oliver  if  the  GAO  had  already 
set  up  the  OMB/GAO  information  system  he  described.   He  replied  that 
the  system  is  not  yet  operational  because  of  understaf f ing.   However 
the  agency  asked  for  new  positions  for  this  operation  in  its  fiscal 
year  1976  budget. 
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'THE  OFFICE  OF  TECHNOLOGY  ASSESSMENT:   A  DESCRIPTION  OF  THE  OFFICE'S 

OBJECTIVES  AND  ACTIVITIES:,  GARY  THOMAS,  OFFICE  OF 

TECHNOLOGY  ASSESSMENT 


The  Technology  Assessment  Act  of  1974  (P.L.  92-484)  authorized  the 
establishment  of  an  Office  of  Technology  Assessment  for  the  Congress  as 
an  aid  to  the  identification  and  consideration  of  existing  and  probable  im- 
pacts of  technological  application. 

In  the  legislation  the  Congress  recognized  that  neither  it  nor  the 
Federal  agencies  were  designed  to  provide  the  Legislative  Branch  with  ade- 
quate and  timely  information  independently  developed  relating  to  the  po- 
tential impact  of  technological  application.   The  Office  of  Technology 
Assessment  was  therefore  created  to  secure  competent,  unbiased  information 
concerning  the  physical,  biological,  economic,  social,  and  political  ef- 
fects of  the  increasingly  extensive  and  larger  applications  of  technology'. 
This  information  is  then  to  be  used  as  one  factor  in  the  decision-making 
process  in  the  Legislative  Branch,  particularly  in  those  areas  where  Con- 
gress must  reach  judgments  on  technological  applications  and  issues  having 
technological  implications. 

As  created  by  the  legislation,  the  Office  of  Technology  Assessment  (OT/ 
is  an  agency  within  and  responsible  to  the  Legislative  Branch  of  the  Govern- 
ment.  Prior  to  the  establishment  of  OTA,  the  General  Accounting  Office 
was  the  last  legislative  office  created  by  the  Congress;  it  was  established 
in  1921.   Since  the  establishment  of  OTA,  a  fourth  legislative  agency,   the 
Congressional  Budget  Office,  has  been  established. 

The  basic  function  of  OTA  is  to  provide,,  on  a  timely  and  independent 
basis,  "early  indications  of  the  probable  beneficial  and  adverse  impacts 
of  the  applications  of  technology  and  to  develop  other  coordinate  informatic 
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which  may  assist  the  Congress."  Eight  specific  activities  are  listed  be- 
low which  are  identified  in  the  legislation  as  the  means  of  carrying  out 
this  function.  The  Office  is  required  to: 

(1)  Identify  existing  or  probable  impacts  of  technology  or 
technological  programs; 

(2)  where  possible,  ascertain  cause-and-eff ect  relationships; 

(3)  identify  alternative  technological  methods  of  implement- 
ing specific  programs; 

(4)  identify  alternative  programs  for  achieving  requisite  goals; 

(5)  make  estimates  and  comparisons  of  the  impacts  of  alternative 
methods  and  programs; 

(6)  present  findings  of  completed  analyses  to  the  appropriate 
legislative  authorities; 

(7)  identify  areas  where  additional  research  or  data  collection 
is  required  to  provide  adequate  support  for  the  assessments 
and  estimates  described  above; 

(8)  undertake  such  additional  associated  activities  as  the  ap- 
propriate authorities  may  direct. 

The  OTA  statute  specifies  that  OTA  shall  consist  of  a  non-partisan 
Congressional  policy  Board,  an  OTA  Director  and  Deputy  Director,  a  cit- 
izens Advisory  Council,  and  such  other  employees  and  consultants  as  may 
be  necessary  in  the  conduct  of  OTA's  work.  The  Congressional  Board  sets 
the  policies  of  the  Office  and  is  the  sole  and  exclusive  oversight  body 
governing  OTA.  The  OTA  Director  is  the  chief  executive  officer  and  is 
responsible  solely  to  the  Board,  of  which  he  is  a  member.   The  function 
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of  the  Advisory  Council  is  to  advise  on  such  technology  assessment  matters 
as  may  be  requested  by  the  Congressional  Board. 

OTA's  Congressional  Board  comprises  six  Senators  and  six  Represen- 
tatives, evenly  divided  by  party,  who  are  appointed  respectively  by  the 
President  Pro  Tempore  of  the  Senate  and  the  Speaker  of  the  House.   The 
current  Board  Chairman  is  Congressman  Olin  E.  Teague  of  Texas  and  the 
Vice  Chairman  is  Senator  Clifford  P.  Case  of  New  Jersey.   The  two  posts 
rotate  between  the  SEnate  and  House  in  alternate  Congresses  with  the 
chairman  chosen  from  the  majority  party,  and  the  vice  chairman  chosen 
from  the  minority  party. 

The  Director  of  OTA  is  Emilio  Q.  Daddario,  a  former  member  of  Con- 
gress who  was  instrumental  in  the  development  of  the  Technology  Assess- 
ment Act.   The  Deputy  Director  is  Daniel  De  Simone,  a  former  White  House 
science  policy  assistant.   The  Chairman  of  the  citizens  advisory  council 
is  Harold  Brown,  President  of  the  California  Institute  of  Technology. 
The  Vice  Charman  is  Edward  Wenk,  Jr.,  of  the  University  of  Washington. 

There  have  also  been  established  for  each  of  the  assessment  programs 
underway  Advisory  Committees  of  distinguished  and  knowledgeable  experts 
regarding  the  specific  issues  being  addressed  by  OTA  for  the  Congress. 

*  *  *  * 

When  the  Technology  Assessment  Board  met  in  early  1974  to  begin  its 
work,  having  previously  contacted  each  of  the  Congressional  Committees, 
it  found  a  number  of  priority  issues  before  it.   The  following  is  a  list 
of  illustrative  topics  that  emerged  at  that  time: 
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Energy  Policy  — 

Impact  of  energy  costs  on  food  production  and  technological  strategies. 

An  assessment  of  the  implications  of  existing  projections  of  energy 
resources  and  requirements;  feasible  and  forecasted  technologies  compared 
with  these  energy  projections. 

Opportunities  for  conservation  of  energy;  evaluation  of  the  trade- 
offs among  different  kinds  of  conservation,  and  between  conservation  and 
expanded  production  capacity. 

Implications  of  large  tankers,  deep  water  ports,  large  storage  capa- 
city, and  related  technologies. 

Alternatives  to  gasoline  for  automotive  power  plants:   hydrogen, 
methane,  methanol,  and  other  possibilities. 

Off-shore  drilling  for  gas  and  oil  and  alternative  sources  of  hydro- 
carbon fuels:   e.g.,  oil  shale,  tar  sands,  coal  gasification. 

Growth  Policy  — 

Potential  effects  of  modern  technologies  —  e.g.,  agricultural,  in- 
dustrial, sewage  treatment,  communications,  law  enforcement,  transporta- 
tion —  on  population  distribution  and  the  growth  of  communities. 

Competing  uses  of  land,  including  technical  assessment  of  the  im- 
plications of  social  values  in  conflict. 

Natural  Resources  — 

Technological  opportunities  for  maximizing  materials  utilization: 
sustained  growth  in  economic  value  of  output  with  reduced  consumption  of 
materials. 
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Recycling  —  the  implications  of  the  closed  cycle  of  materials  usage. 

Substitution  of  materials  in  the  context  of  resources/requirements 
forecasts. 

The  management  of  all  forms  of  materials  information. 

Greater  use  of  abundant  materials  in  energy- limited  industrial  systems. 

The  use  of  renewable  resources  for  synthetic  polymer;  biomass  as  a 
new  industrial  raw  material. 

The  management  of  water:   ground  water,  surface  water,  and  induced 
rainfall;  purification  and  desalting  versus  cascading  of  uses. 

The  technological,  economic,  and  political  implications  of  the  Earth 
Resources  Satellite  System:   dissemination,  management,  and  interpretation 
of  data  therefrom. 

The  technological  aspects  of  foreign  policy  planning  for  the  3rd  Law 
of  the  Sea  Conference. 

Hydrocarbons  as  alternative  sources  of  protein  for  human  and  stock 
food. 

Urban  Development  and  Transportation  — 

Technologies  for  rapid,  large-scale  tunneling  and  excavation:   their 
potential  application  and  social  effects. 

Implications  for  CATV  and  the  "wired  city"  concept. 

Mobile  home  technology  and  its  effects  on  public  health,  safety,  util- 
ity services,  education,  security,  local  tax  revenues,  social  and  munici- 
pal costs  (an  example  of  "low"  technology  development  with  wide-ranging 
consequences). 
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Implications  of  possible  shifts  to  various  forms  of  mass  public 
transportation,  intra-urban  and  inter-urban 

Municipal  sewerage:  disposal  versus  utilization.  The  related  prob- 
lem of  animal  waste  management,  disposal,  and  utilization. 

Noise  control  in  high  density  areas;  trucks  and  airports,  industrial, 
heavy  machinery,  etc. 

Health  Policy  — 

The  increasing  cost  of  health  care  and  delivery;  opportunities  for 
technological  innovation  in  health  care  delivery  systems  to  reduce  costs 
and  improve  services. 

Adequacy,  limitation,  and  utilization  of  scientific  advice  in  the 
establishment  of  safety,  environmental,  and  health  standards. 

Trade  Policy  — 

Effect  of  international  standards,  measurement  practices,  and  certi- 
fication procedures  on  U.  S.  trade. 

Technology  as  the  basis  for  U.  S.  competitiveness  in  international 
trade. 

Trade  implications  of  U.  S.  support  for  the  development  of  regional 
technological  systems  (e.g.,  Mekong,  Rio  Bermejo,  Northeast  Brazil). 

Because  OTA  was  just  beginning  its  work  and  had  limited  means  at  its 
disposal,  it  had  to  be  discriminating  in  the  selection  of  its  activities. 
To  accomplish  this,  the  following  criteria  were  used: 

o  Can  OTA  make  a  unique  contribution,  or  is  it 
within  the  normal  capabilities  of  a  committee 
of  the  Congress  to  perform  this  assessment? 
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o  What  is  the  magnitude  of  the  alleged  or 

expected  costs  and  benefits  to  society  of  the 
various  technological  options  involved  and  how 
are  these  costs  and  benefits  distributed? 
o  Is  the  technological  impact  irreversible? 
o  How  imminent  is  this  impact? 
o  Do  we  know  enough  to  perform  the  assessment? 
o   Is  it  of  manageable  scope? 
o  What  would  the  assessment  cost? 
o  How  long  would  it  take? 
o  What  is  the  likelihood  of  Congressional 
action? 
The  attached  matrix  shows  most  of  the  requests  we  have  received  and 
the  actions  taken  thus  far.  An  assessment  of  drug  bioequivalence  has 
been  completed  and  other  on  food,  solar  energy,  urban  mass  transportation, 
the  extraction  of  oil  and  gas  off  the  coastal  United  States,  industrial 
materials,  super-tankers  and  super-ports,  health,  and  international  trade 
are  all  in  various  stages  of  development. 

To  illustrate  the  process  of  technology  assessment  and  how  the  as- 
sessment will  assist  the  Congress  in  resolving  issues  with  a  technical 
content,  let  us  consider  one  of  the  OTA  assessments  programs  in  more  de- 
tail.  The  specific  current  assessment  programs  include  energy,  food, 
health,  materials,  oceans,  transportation,  technology  and  world  trade, 
and  exploratory  assessments.   The  materials  program  has  several  assess- 
ments in  progress  simultaneously  each  of  which  is  responsive  to  specific 
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Congressional  requests.  The  materials  assessments  are  related,  however, 
and  have  been  designed  to  collectively  develop  an  overall  materials  pro- 
gram. We  shall  therefore  choose  the  materials  program  for  detailed  dis- 
cussion. 

The  OTA  materials  program  was  initiated  by  a  study  of  major  issues 
in  national  policy,  as  already  identified  by  major  study  groups  of  na- 
tional stature,  and  of  major  legislative  interests  in  materials.   The 
materials  science  and  engineering  community  has  long  recognized  that  the 
cycle  of  materials  usage,  including  the  associated  use  of  energy  and  en- 
vironmental impacts,  is  central  to  civilization.   Let  us  briefly  consider 
this  cycle  because  it  is  a  fundamental  concept  to  the  OTA  materials  pro- 
gram.  Materials  flow  through  successive  stages  of  production,  use  and 
disposal,  creating  an  interrelatedness  of  raw  materials,  bulk  materials, 
engineering  materials,  products,  and  waste.   Some  feeling  for  the  size 
of  materials  flow  in  the  United  States  is  given  by  recognizing  that  the 
use  per  person  per  year  is  a  total  of  42,500  lbs.  composed  of  20,550  lbs. 
of  nonmetallic  materials,  17,300  lbs.  of  mineral  fuels,  1,340  lbs.  of 
metals  and  2,310  lbs.  of  organics. 

To  supply  this  cycle  in  1973  the  United  States  imported  $19  billion 
worth  of  minerals  and  exported  $11  billion  worth.   The  nations  of  the 
world  are  inextricably  tied  together  by  the  flow  of  materials.   In  the 
words  of  Edmundo  de  Alba,  they  are  "condemned  to  interdependence".   Con- 
servation of  materials  through  more  effective  utilization  and  other  means, 
such  as  recycling, will  probably  become  increasingly  urgent  but  will  have 
far-reaching  direct  and  indirect  consequences  on  the  economy,  employment, 
etc. 
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Twenty  two  years  ago  the  Paley  report  drew  attention  to  the  enor- 
mously increasing  use  of  mineral  raw  materials,  to  the  problem  of  meeting 
demand  in  the  future,  and  to  the  need  for  conservation  of  materials.   The 
world-wide  boom  of  the  60 's  and  70' s,  fueled  by  inexpensive  petroleum, 
obscured  this  underlying  need.   Discussion  of  the  need  for  conservation 
and  effective  utilization  of  materials  did  continue  but  without  the  ur- 
gency of  immediate  shortages.  Urgency  began  to  develop  in  the  early  70' s, 
however,  as  shortages  began  to  occur  and  rose  to  a  high  level  with  the 
occurrency  of  the  OPEC  petroleum  embargo  and  the  threat  of  similar  ac- 
tions for  other  materials.   Thus,  by  mid-1973  there  was  widespread  rec- 
ognition that  the  United  States  had  a  general,  serious  and  long-term 
problem  with  supplies  of  essential  fuel  materials  and  other  raw  materials. 
This  problem  was  felt  most  acutely  in  the  case  of  the  absolute,  though 
temporary,  shortage  of  oil,  but  it  was  felt  also  in  the  form  of  higher 
prices  for  other  raw  materials.   Considering  the  total  materials  cycle 
again  we  would  expect  shortage  in  fuel  or  other  materials  at  the  begin- 
ning of  the  cycle  to  have  consequences  throughout  the  cycle  as  has  indeed 
happened.   These  intermediate  stage  shortages  are  not  entirely  caused  by 
raw  materials  shortages  but  also  by  limited  productive  capacity  which  in 
turn  is  tied  to  economic  gactors.   The  entire  system  is  under  pressure 
from  the  rising  cost  of  energy  and  theneed  to  reduce  pollution.   Still 
another  pressure  on  the  system  is  increasingly  felt  at  the  disposal  end 
because  availability  of  land  for  waste  disposal  is  decreasing  and  the 
cost  of  disposal  is  increasing.   Indeed,  this  is  one  of  the  principal 
driving  forces  for  increased  recycling.   The  question  of  reconciling  the 
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need  for  accessibility  to  minerals  with  other  uses  of  land  and  for  pre- 
servation of  the  environment  puts  still  another  pressure  on  the  system. 

In  developing  a  program  of  technology  assessments  on  materials,  the 
Office  of  Technology  Assessment  has  proceeded  in  stages.   The  first  step 
was  the  development  of  a  prospectus  for  materials  assessments  based  on 
the  study  mentioned  above.   This  prospectus  was  prepared  with  the  assis- 
tance of  the  Congressional  Research  Service.   The  second  step  was  to  have 
the  Materials  prospectus  reviewed  by  the  Technology  Assessment  Advisory 
Council  and  by  the  Materials  Advisory  Committee.   The  committee  members 
are: 


Earl  H.  Beistline 
University  of  Alaska 

Seymour  L.  Blum 
Mitre  Corporation 

James  Boyd 
Materials  Associates 


Harry  H.  Herman,  Jr. 
Consulting  Engineer 

James  A.  Kent 

Michigan  Technological  Univ. 

Hans  H.  Lansberg 
Resources  for  the  Future 


Lloyd  M.  Cooke 
Union  Carbide  Corp. 

Frank  Fernbach 

United  Steelworkers  of  America 


Elburt  F.  Osborn 

Carnegie  Institution  of  Washington 

R.  Talbot  Page 
Resources  for  the  Future 


Edwin  A.  Gee 

E.  I.  De  Pont  de  Nemours  and  Co.,  Inc. 


Nathan  E.  Promisel 
Consultant 


Bruce  Hannay 

Bell  Telephone  Laboratories 

Bruce  Hannon 
University  of  Illinois 

William  J.  Harris,  Jr. 
Association  of  American  Railroads 


Lois  Sharpe 

League  of  Woman  Voters 

George  A.  Watson 
Ferroalloys  Association 

Jack  H.  Westbrook 
General  Electric  Company 


Julius  Harwood 
Ford  Motor  Company 
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Also,  a  conference  on  National  Materials  Policy  was  organized  by 
the  Federation  of  Materials  Societies  to  deal  with  some  of  the  issues 
raised  in  the  Materials  prospectus.   This  conference  was  held  at  Hen- 
niker,  New  Hampshire  in  August  1974.   A  series  of  workshops  reported 
their  recommendations  including: 

-  Improved  management  of  existing  materials  information  systems; 

-  The  need  for  a  national  stockpile  of  materials; 

-  Economic  and  institutional  barriers  to  innovation  should  be 
removed; 

-  Motivation  should  be  provided  to  encourage  materials  conserva- 
tion in  the  design  stage; 

-  Barriers  to  materials  recycling  should  be  overcome  by  strong 
economic  incentives  and  effective  institutional  arrangements; 

-  Some  materials  goals  are  shared  by  all  countries  and  should 
provide  the  basis  for  concerted  international  program; 

-  There  was  a  need  for  a  permanent  national  materials  commission; 

-  Consideration  should  be  given  to  organizing  a  World  Materials 
Year. 

The  third  step  was  to  have  a  Materials  Committee  adopt  a  set  of 
goals  for  the  overall  strategy  of  the  materials  assessments  program  and 
develop  a  recommended  list  of  individual  assessments.   The  Committee  de- 
cided to  accept  the  goals  of  the  National  Commission  on  Materials  Pol- 
icy as  a  broad  strategy.   These  goals  are: 

(1)  Provide  adequate  energy  and  materials  supplies  to  satisfy 
not  only  the  basic  needs  of  nutrition,  shelter  and  health, 
but  a  dynamic  economy  without  indulgence  in  waste.   The 
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other  general  goals  of  the  NCMP  support  this  primary  goal 
and  suggest  a  framework  of  policy  constraints. 

(2)  Rely  on  market  forces  as  a  prime  determinant  of  the  mix  of 
imports  and  domestic  production  in  the  field  of  materials 
but  at  the  same  time  decrease  and  prevent  wherever  necessary 
a  dangerous  and  costly  dependence  on  imports. 

(3)  Accomplish  the  foregoing  objectives  while  protecting  or 
enhancing  the  environment  in  which  we  live. 

(4)  Conserve  our  material  resources  and  environment  by  treating 
waste  materials  as  resources  and  returning  them  either  to 
use  or,  in  a  harmless  condition,  to  the  ecosystems. 

(5)  Institute  coordinated  resource  policy  planning  which  recog- 
nizes the  interrelationships  among  materials,  energy,  and 
the  environment. 

With  these  goals  in  mind  and  with  the  entire  materials  cycle  under 
consideration,  the  materials  advisory  committee  recommended  the  follow- 
ing list  of  assessments  on  September  20,  1974: 

(1)  Devise  materials  information  systems  for  reliable  input  to 
policymaking. 

(2)  Establish  a  national  stockpile  policy. 

(3)  Stimulate  efforts  to  hasten  materials  recycling. 

(4)  Develop  means  to  encourage  and  assist  manufacturing  industries 
to  us  materials  in  fabricating  products  employing  materials 
more  effectively. 
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(5)  Manage  materials  so  as  to  conserve  energy,  but  in  a  manner 
to  minimize  economic  and  social  dislocation. 

(6)  Expand  and  strengthen  domestic  minerals  industry. 

(7)  Stimulate  education,  research  and  development  in  the  mineral 
extraction  and  processing  fields. 

(8)  Assess  the  interaction  of  environmental  concerns  with  effec- 
tive utilization  and  production  of  materials. 

(9)  Manage  materials  so  as  to  conserve  materials,  but  in  a  man- 
ner to  minimize  economic  and  social  dislocation. 

(10)   Examine  land  use  in  relation  to  laws  regarding  mineral  ex- 
ploration and  production. 
Assessments  on  materials  information  systems  and  on  national  stock- 
pile policy  are  now  underway.  Assessments  are  also  being  designed  in 
the  areas  of:   a)  resource  recovery,  materials  recycling  and  reuse,  and 

b)  institutional  constraints  on  domestic  mineral  accessibility  and  c) 

c)  conservation  of  materials.  Further  assessments  on  other  aspects  of 
the  cycle  of  materials  use  are  under  consideration. 

These  assessments  were  developed  in  response  to  Congressional  re- 
quests.  These  requests  included  not  only  the  first  set  received  early 
in  1974,  but  also  a  later  set  which  were  received  after  the  Materials 
Advisory  Committee's  recommendations  and  dreq  upon  these  recommendations. 
These  latter  were  more  specific  requests  within  the  broad  areas  described 
in  the  first  cycle  of  requests  for  materials  assessments. 

The  House  Science  and  Astronautics  Committee,  which  has  since  be- 
come the  House  Science  and  Technology  Committee,  made  a  broad  request. 
In  their  first  letter,  dated  January  22,  1974,  Chairman  Teague  and 
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ranking  minority  member  Mosher  emphasized  four  areas  for  possible  OTA 
study,  including  a  technological  data  base  for  Congress  and  research  and 
development  programs  to  lessen  United  States  dependence  on  importation 
of  critical  materials.   In  their  second  letter,  dated  December  13,  1974, 
they  specifically  requested  studies  of  1)  materials  information  systems, 
2)  national  stockpile  options,  and  3)  reuse  of  materials. 

The  Senate  Commerce  Committee,  in  a  letter  from  Senator  Magnuson 
dated  January  24,  1974,  requested  studies  of  the  solid  waste  problem 
including  1)  reduction  of  waste  at  the  source,  2)  recycling  and  resource 
recovery,  and  3)  energy  recpvery. 

Representative  Morris  Udall,  a  member  of  the  Technology  Assessment 
Board,  in  a  letter  dated  September  19,  1974,  raised  several  questions 
to  be  answered  by  assessments: 

"What  means  do  we  have  to  deal  with  impending  resource 
scarcities?  What  kind  of  roles  will  such  methods  as 
substitution  of  new  materials  for  scarce  ones,  rationing, 
altered  pricing  systems,  reuse  and  recycling,  new  effi- 
cient production  technologies  and  new  regulations  gov- 
erning land  use  play  under  these  conditions?" 
Senator  Ted  Stevens,  a  member  of  the  Technology  Assessment  Board, 
requested  a  study  of  mineral  accessibility  in  a  letter  dated  November  6, 
1974. 

In  addition  to  the  interest  in  materials  shown  by  these  specific 
requests  to  OTA,  there  is  widespread  Congressional  interest  in  materials. 
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As  Chairman  of  the  Senate  Commerce  Committee,  Senator  Warren 
Magnuson  in  a  letter  dated  January  15,  1975,  endorsed  a  request  by 
Senator  John  Tunney  for  a  study  of  means  of  conserving  materials 
through  reducing  wastage  of  materials  by  reducing  corrosion  and  other 
wastage  processes.   Senator  Magnuson  asked  for:  "1)  an  assessment  of 
the  kinds  and  amounts  of  materials  wastage;  2)  techniques  for  reducing 
wastage;  and  3)  technical  and  institutional  impediments  to  applying 
these  techniques." 

Over  140  separate  bills  on  Materials  were  introduced  into  the  93rd 
Congress  in  the  areas  of  Materials  management,  Materials  recycling, 
Materials  shortages,  and  Materials  stockpiling.   One  bill  which  sub- 
sequently became  law  established  the  National  Commission  on  Supplies 
and  Shortages. 

The  Materials  Information  System  and  National  Stockpile  Assessment 

are  pertinent  to  this  Commission  which  was  established  by  Public  Law 

93-426  on  September  30,  1974.   The  Commission  was  required  to  make 

recommendations,  no  later  than  March  1,  1975, 

"with  respect  to  institutional  adjustments,  including  the 
advisability  of  establishing  an  independent  agency,  to  provide 
for  a  comprehensive  data  and  storage  system,  to  aid  in  examination 
and  analysis  of  the  supplies  and  shortages  in  the  economy  of  the 
United  States  and  the  rest  of  the  world." 

Until  June  30,  1975,  the  Commission  may  make  such  other  reports  and 

recommendations  as  it  thinks  appropriate.   The  Congressional  findings 
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which  led  to  the  establishment  of  the  Commission  were  as  follows: 

1.  The  United  States  is  increasingly  dependent  on  the  Importation 
from  foreign  nations  of  certain  natural  resources  vital  to 
commerce  and  the  national  defense. 

2.  Nations  that  export  s-jch  resources  can  alone  or  in  association 
with  other  notions  arbitrarily  raise,  the  prices  of  such 
resources  to  levels  which  are  unreasonable  and  disruptive 

of  domestic  and  foreign  economics. 

3.  Shortages  of  resources  and  commodities  are  becoming  increas- 
ingly frequent  in  the  United  States,  and  such  shortages  cause 
undue  inconvenience  and  expense  to  con  umers  and  a  burden 

on  interstate  conferee  and  the  Nation  J   cono'my. 

4.  Existing  institutions  do  not  adequately  dentify  and  ar'ici- 
pate  such  shortages  and  do  not  adequately  monitor,  study,  and 
analyze  other  market  adversities  involving  specific  industries 
and  specific  sectors  of  the  economy. 

5.  Data  with  respect  to  such  shortages  and  adversities  is  collected 
in  various  agencies  of  the  Government  for  various  purposes,  but 
is  not  systematically  coordinated  and  disseminated  to  the 
appropriate,  agencies  and  to  the  Congress.  38/  t 

"The  dates  have  apparently  been  changed  to  June  30  and  December  31,  respectively. 

The  OTA  accordingly  proposes  to, assess  pertinent  features  of 

materials  information  systems  in  terms  of  their  past,  present,  and 

expected  stages  of  development.  Major  deficiencies  in  the  existing 

information  systems  should  be  identified  and  alternatives  for  their 


38 /(See  section  720  of  the  Defense  Production  Act  of  1950  as  amended, 
50  U.S.  C.  App.  2169.  The  portions  quoted  above  are  subsections  (h) 
and  (b)  attachment  1). 
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removal  described  and  evaluated.   The  establishment  of  a  materials 
information  system  may  call  for  the  creation  of  a  new  comprehensive 
information  system,  expansion  of  the  present  activities,  or  the  establish- 
ment of  an  institution  charged  with  insuring  efficient  and  effective 
use  of  existing  systems. 

The  law  establishing  the  National  Commission  on  Supplies  and 
Shortages  also  charges  the  Conmi.ssion  to  report 

"necessary  legislative  and  administrative  actions  to  develop 
a  comprehensive  strategic  and  economic  stockpiling  and 
inventories  policy  which  facilitates  the  availability  of 
essential  resources"  (sec.  720  (q)(5)  of  the  Defense 
Production  Act  of  1^50  as  amended;  50  USG  App.  2169.   Under- 
lining supplied). 

Accordingly,  the  OTA  proposes  to  examine  the  consequences  of  alternative 

national  stockpile  policies.   The  possible  uses  of  a  national  stockpile 

for  broader  public  purposes,  in  addition  to  the  limited  national  security 

purposes  for  which  the  "Strategic  and  Critical  Materials  Stockpile"  was 

established,  will  be  assessad. 

/   The  problem  is  to  identify  and  define  the  purposes  that  could  be 

! 

served  by  a  Government-managed  or  Government-financed  institution  that 
bought,  held,  perhaps  upgraded,  and  sold  selected  materials;  to  determine 
the  criteria  governing  the  management  and  operation  of  such  an  institution ; 
to  explore  and  report  on  the  direct  and  indirect  consequences  of  such 
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operation;  to  identify  ways  of  enhancing  favorable  consequences  and 
ways  of  mitigating  or  avoiding  unfavorable  consequences;  and  to  assess 
options  and  alternatives  in  the  design  and  operation  of  various  institu- 
tional arrangements  for  performing  the  function  in  question. 

The  second  assessment,  on  national  stockpile  policy,  will  examine 
the  attributes  and  consequences  of  alternative  national  stockpile 
policies.   The  possible  uses  of  a  national  stockpile  for  broader  pur- 
poses than  the  limited  national  security  purposes  for  which  the  "Stra- 
tegic and  Critical  Materials  Stockpile"  was  established,  will  be 
assessed. 

This  assessment  started  in  March,  1975;  the  final  report  is  due 
in  September,  1975. 

The  third  assessment,  on  resource  recovery,  materials  recycling 
and  reuse,  will  examine  the  institutional  barriers  to,  and  incentive 
for,  achieving  substantial  resource  recovery  from  urban  refuse  using 
the  best  current  technology.   To  accomplish  this  task,  assessments 
of  interrelationships  among  (1)  technology  requirements,  (2)  economics, 
(3)  institutional  barriers  and  incentives,  and  (4)  social  factors  will 
be  made.   The  assessment  will  include  consideration  of  barriers  to 
policy-making  and  to  decision-making  as  well  as  to  operational  imple- 
mentation of  resource  recovery  technology. 

The  fourth  assessment,  on  institutional  constraints  on  domestic 
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mineral  accessibility,  will  consider  the  potential  effects  of  modifying 
the  structure  of  Federal  laws  and  other  institutional  factors  affecting 
the  accessibility  of  domestic  mineral  resources.   This  study  will 
include  consideration  of  all  steps  leading  to  and  including  the  ap- 
plication of  technology  for  the  purposes  of  mineral  exploration, 
development,  extraction,  processing,  and  delivery. 

The  fifth  proposed  assessment,  on  Conservation  of  Materials  through 
reduced  wastage,  will  examine  types  of  materials  wastage  such  as  cor- 
rosion and  wear.   Estimates  of  the  magnitude  of  loss  will  be  made  and 
estimates  of  the  savings  which  could  be  accomplished  through  improved 
design  and  through  application  of  protective  procedures  in  use.   The 
impacts  of  reduced  wastage  will  be  assessed  including  estimated 
impacts  on  the  economy  and  on  employment. 
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ASSESSMENT  REQUESTS  BY  COMMITTEE 

FOR  THE  PERIOD  DKCEMBE*  1973  THROUGH  MARCH  19  75 


House  Committee  on  Agriculture 

2/13/75  Chairman  Foley  requests  assessment  of  feasibility  of  using 
rice-blended  food  products  as  material  eligible  for  export 
under  Title  II  of  the  P.L.  480  program. 

House  Committee  on  Appropriations 

*  11/19/74   Chairman  Mahon,  on  behalf  of  Chairman  McFall  and  Ranking 

Minority  Member  Conte  of  the  Subcommittee  on  Transportation 
Appropriations,  requests  a  technology  assessment  with  regard 
to  automobile  crash  recorders. 

House  Committee  on  Foreign  Affairs 

1/18/74   Chairman  Morgan  suggests  assessments  concerning: 

1)  Arms  control 

2)  Food 

3)  Technology  transfer 

4)  Population/family  planning 

House  Committee  on  Judiciary 

**12/19/7"3  Chairman  Rodino  requests  assessment  of: 

**  1)  Existing  offshore  oil  and  gas  operations 
**  2)  Effects  of  tripling  present  acreage  of  leases  for  oil 
and  gas  production  on  Outer  Continental  Shelf 
3)  Detailed  assessment  of  oil  and  gas  operations  on 
Outer  Continental  Shelf 

House  Committee  on  Merchant  Marine  and  Fisheries 

3/15/74     Chairman  Sullivan,  -jointly  with  'Chairman  Teaeue,  Committee 
on  Science  and  Technology,  express  interest  in  assessment 
of  international  shipping,  including: 

1)  Fundamental  shipping  technology 

2)  Utilization  of  nuclear  technology 

3)  Airfoil  techniques 

4)  Fast  sailing  vessels 

**4/3/74    Chairman  Sullivan  requested  assessments  concerning:  - 

**1)  Exploration  and  exploitation  of  resources  of  Continental 

Shelf  and  deep  seabed  beyond 
**2)  Fisheries 

3)  Aquaculture  and  mariculture 

4)  Oceans  pollution  -  monitoring 

5)  Oceans  pollution  -  removal 

6)  Submersible  vessels 

7)  Undersea  habitats 


*  Assessments  currently  being  performed 
**  Assessments  planned  for  FY  1975  and  1976 
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9/18/74    Chairman  Sullivan  expresses  interest  in  TAAC  National 
Growth  Policy  Study  and  Committee's  deep  interest  in 
long-term  growth  and  its  implications. 

3/7/75   Chairman  Sullivan  and  Ranking  Minority  Member  Ruppe  request 
assessment  of  alternatives  to  unregulated  ocean  dumping  and 
current  status  of  related  research  and  development. 

House  Committee  on  Public" Works 

1/23/74    Chairman  Blatnik  suggests  following  areas  of  interest: 

1)  National  public  investment  policy  and  population 
distribution 

2)  Transportation  policy 

3)  Water  resources 

4)  Pollution  abatement 


House  Coinmitt< 


n  Science  and  Technology 


*••  1/22/74 


3/15/74 


12/13/74 


Cli;:    an  Teague  and  Ranking  Minority  Member  Mosher  request 
ass   iments  concerning: 

1)  Energy  R  6,  D 

2)  Technology    data  bank 

3)  Materials   R&D 

4)  Technology   of   unemployment 

Chairman  Teague,  jointly  with  Chairman  Sullivan  of  the 
Committee  on  Merchant  Marine  and  Fisheries,  expresses 
strong  interest  in  international  shipping,  including: 

1)  Fundamental  shipping  technology 

2)  Utilization  of  nuclear  technology 

3)  Airfoil  techniques 

4)  Fast  sailing  vessels 

Chairman  Teague  and  Ranking  Minority  Member  Mosher    update 
of  previous  request  of  1/22/74,  with  emphasis  on  assess- 
ments involving: 
*  1)  Materials 

*  a)    information   system 

*  b)    national   stockpile 

*  c)    reuse 
#*2)    Five-year   r&d 


12/17/74        Chairman   Teague    and  Ranking  Minority  Member  Mosher   re- 
quest,   or     emphasize      urgency   of,    assessments    involving: 

*  1)    Planning   and  programming  energy  '   R&D     -   as   contained 

in   required   ERDA  reports. 

2)  Identifying  size  of   energy   resources   and   unassociated 
uncertainties 

3)  Materials    technology   and   availability 
*4)    Tracking  energy   studies    and   data 

*  5)    Uncertainties  which    inhibit  development  of  new  energy 

technologies 
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House  Committee  on  Ways  and  Means 

**  1/29/75    Chairman  Ullman,  jointly  with  Ranking  Minority  Member 
Schneebeli  and  Chairman  Rostenkowski  and  Ranking 
Minority  Member  Pettis  of  the  Subcoinmitte  on  Health, 
requests  assessments  concerning: 
**  1)  Medical  malpractice 
**2)  Long-term  medical  care 
3)  Adverse  drug  reactions 

Senate  Committee  on  Aeronautical  and  Space  Sciences 

*  1/21/74   Chairman  Moss,  acting  for  Senator  Goldwater,  requests 

assessment  concerning  solar  energy. 

**  10/10/74'  Chairman  Moss  requests  assessment  of  means  of  determining 
research  and  development  priorities. 

Senate  Commit  tee  on  Agriculture  and  Forrestry 

**6/25/74   Chairman  Talmadge  requests  assessment  of  use  of  broadband 
two-way  telecommunications  in  rural  areas. 

Senate  Committee  on  Appropriations 

*  2/15/74   Chairman  McClellan,  on  behalf  of  Chairman  Byrd  and  Ranking 

Minority  Member  Case  of  the  Transportation  Subcommittee, 
requests  assessments  concerning  automation  in  urban  rapid 
transit . 

*  9/27/74   Chairman  McClellan,  on  behalf  of  Chairman  Byrd  and  Ranking 

Minority  Member  Case  of  the  Transportation  Subcommittee, 
requests  expansion  of  mass  transit  assessment. 

Senate  Committee  on  Commerce 


1/14/74   Senator  Hollings'  National  Oceans  Policy  Study  requests 
assessments  concerning: 

*  1)  Impact  of  technology  on  coastal  zone 
2)  Ocean  engineering  technology 

**  3)  Fisheries 

*  4)  Deepwater  ports 
**  5)  Ocean  mining 

6)  Weather  modification 

7)  Acquaculture 

8)  Oceans  monitoring 
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*  1/24/74   Chairman  Magnuson  suggests  assessments  concerning: 

1)  Energy  savings  in  manufacturing 

2)  Safety  in  disposal  of  nuclear  waste 
*  3)  Resource  and  energy  recovery 

4)  Upgrading  railroad  tracks 
•••  5)  Oceans  technology,  including 
»*  a)  fisheries 
*  b)  deep" water  ports 

6)  Retrofitting  offices  and  residences  for  energy  savings 

7)  Alternative  energy  sources  for  autos 

8)  Mutagenic  testing 

9)  Detergents 

10)  Predator  poisons 

11)  Population  and  conservation  taxes 


2/11/74 


Senator  Rollings'  National  Oceans  Policy  Study  requests 
assessment  on  utilization  of  U.S.  Continental  Shelf. 


*  8/13/74   Chairman  Magnuson,  jointly  with  Chairman  Rollings  of  Sub- 

committee on  Oceans  and  Atmosphere,  request  assessment  of: 
*  1)  National  growth  policy 

*2)  Projected  effect  of  economic  and  demographic  growth  of 
coastal  zone. 

10/10/74  Chairman  Magnuson,  acting  for  Chairman  Tunney  of  the  Sub- 
committee on  Science,  Technology  and  Commerce,  requests 
assessment  of  computer  managed  technology 

'  10/17/74   Chairman  Hart  of  the  Subcommittee  on  the  Environment   re- 
quests assessment  of  cost  and  benefit  of  environmental  pro- 
tection regulations  and  equipment  on  the  automobile  industry, 

*  1/15/75   Chairman  Magnuson,  acting  for  Senator  Tunney,  recommends 

assessment  involving  materials  wastage. 

*  1/23/75   Chairman  Magnuson,  jointly  with  Chairman  Jackson,  Committee 

on  Interior  and  Insular  Affairs,  requests  assessment  of 
feasibility  of  separating  consideration  of  leasing  for 
exploration  from  leasing  for  development  and  production 
on  Outer  Continental  Shelf. 

?*  2/19/75   Chairman  Magnuson,  on  behalf  of  Senator  Tunney,  requests 
comprehensive  assessment  of  technology  and  world  trade. 


3/20/75   Chairman  Magnuson  and  Ranking  minority  Member  Pearson,  jointly 
with  Senators  Hartke  and  Weiker,  Chairman  and  Ranking  Minority 
Member  of  the  Surface  Transportation  Subcommittee,  request 
review  of  U.S.  Railway  Association  plan  for  reorganization 
of  rail  service  in  the  17  state  region  covered  by  the  Regional 
Rail  Reorganization  Act  of  1973,  and  the  issues  it  raises  about 
the  future  of  rail  service  in  this  region 
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| e  Committee  on  Foreign  Relations 

*  9/19/74   Chairman  Fulbright ,  acting  for  Chairman  Muskie  and  Ranking 
Minority  Member  Case  of  the  Subcommittee  on  Arms  Control, 
International  Law  and  Organization,  requests  an  assessment  of  the 
the  accuracy  of  DOD  estimates  of  potential  effects  of  limited 
nuclear  warfare  on  U.S.  society. 

Senate  Committee- on  F inane e 

2/27/75   Chairman  Long,  on  behalf  of  Senator  Talmadge,  Chairman  of  the 
Subcommittee  on  Health,  requests  assessment  of: 

1)  Physician's  fees 

2)  Hospital  Out-Patient  Service 

Senate  Commit  tee 'on  Interior  and  Insular^Afj^-e 

*.  12/30/7/,  ^airman  Jackson  indicates  need  for  comprehensive  assessment 
of  energy  technology  and  joins  request  of  House  Committee 


on  Science  and  Technology  for  that  purpos 


1/23/75   Chairman  Jackson,  jointly  with  Chairman  Magnuson,  Committee 
on  Commerce,  requests  assessment  of  feasibility  of  separati, 
consideration  of  leasing  for  exploration  from  leasing  for  ' 
development  and  production  on  Outer  Continental  Shelf. 

Senate  Committee  on  Labor  &  Public  Welfare 

**12/ll/74  Chairman  Williams,  jointly  with  Senator  Javits  and  Congress- 
men Rogers,  Nelsen,  and  Carter  of  the  House  Subcommittee  on 
Public  Health  and  Environment,  Committee  on  Interstate  and 
Foreign  Commerce,  request  assessments  in: 
**1)  Spiraling  costs  of  health  care 
2)  Uneven  quality  of  health  care 

**  2/6/75   Chairman  Williams  ,  on  behalf  of  Chairman  Kennedy  and  Ranking 
Minority  Member  Javits  of  the  "Health  Subcommittee  ,  requests 
technology  assessments  on  the  following: 
**1)  Cost  and  quality  of  clinical  laboratories 

2)  Medical  record  information  requirements 
**3)  Cost  control  studies,  i.e.,  effect  of  regulation  of 
price,  effect  of  deductibles  and  coinsurance  on  ut- 
ilization of  health  care,  efficacy  of  new  technology 
and  procedures,  productivity  measures,  and  cost  of  ad- 
ministering health  insurance. 

Senate  Committee  on  Public  VJorks 

1/29/75     Chairman  Randolph  requests  assessment  of  Federal  assistance 
to  energy  and  coal  research  facilities  in  Appalachia  and 
West  Virginia. 

Senate  Select  Committee  on  Nutrition  and  Human  Needs 

**  2/7/75  Chairman  McGovern  asks  OTA  to  determine  whether  protein  is 
being  wasted  by  being  fed  as  grain  to  livestock,  and  if  so, 
what  government  policy  changes  are  necessary  to  remedy  this 
waste. 
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Joint  Economic  Commit' tee 

9/10/74    Chairman  Patman,  acting  for  Chairman  Bentsen  of  the 
Subcommittee  on  Economic  Growth,  requests  assessment 
of  industrial  innovation  in  enhancing  productivity  and 
constraining  prices  with  respect  particularly  to: 

1)  Housing 

2)  Power  generation 

3)  Transportation 

4)  Basic  metals  and  machine  tools 

2/11/75     Chairman  Humphrey  requests  assessment  of  feasibility  of  im- 
proving and  enlarging  defense  research  and  production  facilities 

'■'■'  2/28/75   Chairman  Humphrey  and  Representative  Reuss,  Chairman  of  the 
Subcommittee  on  International  Economics,  request  a  compre- 
hensive study  of  technology  and  world  trade  with  assessments 
of  policy  options  to  strengthen  international  trade  position. 


Technology  Assessment  Board 

*l/22/74    Senator  Humphrey  suggests  assessments  concerning: 
*1)  Agricultural  information 

a)  use  of  ERT  satellites  in  agriculture 

2)  Fertilizer 

3)  Food  processing  and  storage 

4)  Farm-to-market  roads 

*  9/19/74    Congressman  Udall  suggests  assessment  of  a  proposed 

National  Growth  Policy. 

*  11/6/74    Senator  Stevens  suggests  topic  of  accessibility  of 

mineral  resources  for  assessment. 

*  11/7/74    Senator  Humphrey  suggests  assessment  of  priorities  in 

federally  supported  research  and  development. 

2/26/74   Senator  Schweiker  requests  review  of  U.S.  Railway  Association's 
conrail  plan.  -  ; 


Other  Members  of  the  Congress 

1/7/74     Congressman  Wyman  recommends  for  consideration  Federally- 
assisted  private  corporations  to  develop  a  commercially 
feasible  process  for. making  gas  and  oil  from  coal. 

1/28/74    Senator  Hartke  proposes  consideration  of  impact  of  tech- 
nology on  future  growth  and  development  of  the  Nation. 

1/28/74    Congressman  Vanik  requests  data  concerning  reduced  energy 
consumption  attributable  to  winter  Daylight  Savings  Time. 

1/29/74    Congressman  Perkins  suggests  the  problem  of  outdated  statis- 
tical information  as  area  of  concern. 
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2/14/74    Congressman  Whit ten,  making  no  specific  request,  callu 
attention  to  work  in  the  following  areas: 

1)  Food  and  Drug  assessment  of  need  for  Delaney  clause 

2)  Study  of  future  lab  needs  for  Food  and  Drug 

3)  Appropriation  to  National  Academy  of  Sciences  to 
evaluate  programs  of  Environmental  Protection  Agency. 


11/21/74   Congressman  Aspin  suggests  as  area  for 
gers  involved  in  use  of  aerosols. 


issessment   the  dan- 


12/3/74 


Congressman  Pickle  requests  study  of  dangers  imposed  by 
use  of  freon  in  aerosol  cans. 


2/10/75   Congressman  Wolfe  requests  analysis  of  study  made  on  atmosphci 
effects  of  a  fleet  of  supersonic  transports. 
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October  13,  1972  -  7  -  Pub.  Law  92-484 


t6  STAT.  *03 


Sac.  12.  (e)  To  enable  the  Oflce  to  carry  out  ita  power*  end  duties, 
there  ie  hereby  authorised  to  be  appropriated  to  the  OAce,  out  of  any 
v  in  the  Treasury  not  otherwise  expropriated,  not  to  exceed 
),000  in  the  aggregate  for  the  two  fiscal  yean  ending  Jane  30, 
,  end  Jnne  80,  1974,  and  thereafter  wen  sum*  a*  may  be  necessary, 
(b )  Appropriation*  made  pursuant  to  the  authority  provided  in 
•ubeectkm  (a)  ahall  remain  evailable  for  obligation,  for  expendi- 
ture, or  for  obligation  and  expenditure  for  each  period  or  period*  ee 
may  be  epednedin  the  Act  making  eneh  appropriation* 
Approved  October  13,  1972. 


LtGISUTIVI  HIS  TORY  I 

HOUSE  REPORTS i    *.  92-4W  (com.  ee  Science  and  Astrenautlea)  and 

Re.  92-1436  (Cam.  of  Conference). 
SENATE  REPORT      No.  92-1123  (Cobb,  on  Role*  and  Ada IMrt rati ea). 
CORORESSIOMAL  RECORD,   fel.   11*   (1872).  u»-cra*ienj. 

Peb.    B,  oencldered  and  paased  House. 

Sept. 14,  eonaldered  and  paaaed  Senate,  aaaaded. 

Sept. 22,  Senate  agreed  to  conference  report. 

Oct.    4,  Houcc  agreed  to  conference  report. 
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October  13,  1972  -  7  -  Pub.  Law  92-484 


86  STAT.    803 


APPROPRIATIONS 

Sec.  12.  (a)  To  enable  the  Office  to  carry  out  its  powers  and  duties, 
there  is  hereby  authorized  to  be  appropriated  to  the  Office,  out  of  any 
money  in  the  Treasury  not  otherwise  appropriated,  not  to  exceed 
$5,000,000  in  the  aggregate  for  the  two  fiscal  years  ending  June  30, 
1973,  and  June  30, 1974,  and  thereafter  such  sums  as  may  be  necessary. 

(b)  Appropriations  made  pursuant  to  the  authority  provided  in 
subsection  (a)  shall  remain  available  for  obligation,  for  expendi- 
ture, or  for  obligation  and  expenditure  for  such  period  or  periods  as 
may  be  specified  in  the  Act  making  such  appropriations. 

Approved  October  13,  1972. 


LEGISLATIVE  HISTORY; 

HOUSE  REPORTS*     No.  92-469   (Conn,   on  Scienoe  and  Astronautics)  and 

No.  92-1436  (Conn,   of  Conferenoe). 
SENATE  REPORT       No.  92-1123  (Conn,   on  Rules  and  Administration). 
CONGRESSIONAL  RECORD,    Vol.    118    (1972) i 

Feb.     8,   considered  and  passed  House. 

Sept. 14,   oonsidered  and  passed  Senate,   amended. 

Sept. 22,  Senate  agreed  to  oonferenoe  report. 

Oot,     4,    House  agreed  to  oonferenoe  report. 
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DISCUSSION 

Several  issues  were  raised  after  Dr.  Thomas'  presentation.   It  was 
noted  that  both  its  legislative  mandate  and  legislative  exigencies  require 
the  OTA's  work  to  be  especially  timely  and  responsive  to  the  Congress.   In 

addition,  the  OTA  maintains  very  close  liaison  with  committees  both  to 

receive  suggestions  for  assessments,  and  to  brief  committee  staff  through- 
out the  process  of  the  assessment,  making  available  working  documents  and 
monthly  reports  submitted  by  contractors. 

Mr.  Thomas  also  noted  that  the  OTA  does  two  types  of  assessments  

both  long  and  short  range.  Short  range  or  "mini-assessments"  are  done  on 
an  urgent  basis  by  the  exploratory  assessment  section  of  the  Office.  For 
example  the  OTA  recently  completed  a  two  week  assessment  of  the  impact  of 
the  ERDA  budget.  The  exploratory  assessment  section  serves  another  function 
in  addition  to  that  of  providing  rapid  responses  on  assessment  questions. 
It  performs  exploratory  assessments  to  determine  whether  an  in-depth. long- 
range  assessment  is  warranted. 

A  question  was  raised  about  the  possible  similarities  between  technology 
assessment  research  and  science  policy  research.   Dr.  Thomas  remarked  that 
the  two  clearly  are  distinct.   Technology  assessments  generate  information 
to  be  used  in  policy  research  or  policy  deliberations.   As  an  example  he 
noted  that  the  assessment  of  materials  information  system  is  designed  to 
generate  information  to  indicate  where  such  information  might  be  used  most 
appropriately  in  policy  research. 
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["OVERSIGHT  RESPONSIBILITIES  AND  ACTIVITIES  OF  THE  SENATE  COMMITTEE  ON 

GOVERNMENT  OPERATIONS  AND  SEVERAL  OF  ITS  SUBCOMMITTEES,"]  STAFF  OF  THE 

SENATE  COMMITTEE  ON  GOVERNMENT  OPERATIONS:   STATEMENTS  BY  MARILYN 

HARRIS,  HOWARD  FELDMAN,  JIM  DAVIDSON,  BILL  GOODWIN,  VIC  REINEMER 

AND  RON  CHIODO 


Ms.  Marilyn  Harris 

I  believe  Mr.  Beckman  says  it  well.   Our  Committee  has  a  tremendous 
challenge  which  is  the  oversight  of  the  agencies  of  the  government,  espe- 
cially with  respect  to  their  economy  and  efficiency.   As  you  know  Senator 
Ervin  was  Chairman  of  the  Government  Operations  Committee  until  his  re- 
tirement.  This  year,  Senator  Ribicoff  became  Chairman.   The  committee 
reorganized  its  subcommittees  and  changed  some  jurisdictions  with  the 
hope,  naturally,  of  making  our  Committee  as  efficient  as  we  would  like 
the  Federal  Government  to  be  also.   As  part  of  our  organization,  we  estab- 
lished the  Oversight  Procedures  Subcommittee,  about  which  Mr.  Goodwin  will 
be  speaking.   One  Subcommittee  was  abolished,  Reorganization,  Research  and 
International  Organizations  and  that  jurisdiction  was  taken  to  the  full  Com- 
mittee.  Our  Committee  is  organized  into  unique  Subcommittees.   Each  Sub- 
committee has  its  own  budget  and  is  funded  under  a  separate  section  of  a 
General  Funding  Resolution.   Because  of  this  type  of  organization,  there 
is  quite  a  bit  of  autonomy  within  our  subcommittees,  whereas  in  some  other 
committees,  there  may  not  be  as  much  autonomy  because  of  the  type  of  com- 
mittee structure  established. 

I  wanted  to  introduce  the  panel  to  you,  and  I  thought  the  best  way  for 
you  to  achieve  some  understanding  of  our  Committee  and  its  Subcommittees  is 
for  each  person  who  is  representing  that  subcommittee  to  tell  you  specifically 
what  it  is  doing.   So  I  will  introduce  the  panel  to  you  with  that  objective 
in  mind. 
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On  my  left  is  Jim  Davidson,  Counsel  to  the  Intergovernmental  Relations 
Subcommittee.   That  Subcommittee  investigates  the  intergovernmental  rela- 
tions between  the  U.S.  and  the  states  and  municipalities,  including  the 
fiscal  relationship  between  the  state,  local  and  Federal  governments.  Jim 
will  tell  you  more  when  we  begin.   On  my  immediate  right  is  Mr.  Howard 
Feldman,  who  is  the  General  Counsel  to  the  Permanent  Subcommittee  on  In- 
vestigations.  This  Subcommittee  is  non-legislative  and  is  concerned  strictly 
with  investigation  and  oversight.   And  to  his  right  is  Mr.  Bill  Goodwin  the 
staff  director  of  our  newly  created  Subcommittee,  the  Oversight  Procedures 
Subcommittee.   One  of  its  main  goals  will  be  to  develop  methods  to  improve 
how  the  Government  Operations  Committee  and  other  committees  can  develop 
procedures  for  improving  oversight.   Also,  Vic  Reinemer,  staff  director  of 
the  Reports,  Accounting  and  Management  Subcommittee* Senator  Metcalf  that 
Subcommittee's  Chairman,  was  one  of  the  Senators  who,  I  believe,  generated 
the  initiation  of  this  Conference.   Last  year,  the  Reports  Subcommittee 
was   titled  the  Budgeting  and  Management  Subcommittee.   Some  of  that  Sub- 
committee's jurisdiction  was  changed.   It  is  concerned  with  the  accounting, 
financial  reporting  and  auditing  arrangements  and  the  development  of  manage- 
ment capability  and  efficiency  of  the  Federal  Government.   It  is  also  con- 
cerned with  Federal  advisory  committees,  Federal  reports,  questionnaires  of  the 
Federal  Government  and  interrogatories.   On  the  very  end  is  Mr.  Ron  Chiodo, 
a  counsel  from  the  Subcommittee  on  Federal  Spending  Practices,  Efficiency 
and  Open  Government.   This  Subcommittee  is  chaired  by  Senator  Chiles.   Last 
year  it  was  called  the  Subcommittee  or  the  Ad  Hoc  Subcommittee  on  Procure- 
ment, and  has  now  been  changed  to  a  permanent  subcommittee. 

I  think  we  will  begin.   Mr.  Feldman. 
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Mr.  Howard  Feldman 

Thank  you.   I  think  our  Subcommittee  is  probably  unique.   I  must  take 
issue  with  the  comment  that  we  don't  have  a  legislative  purpose.   Actually 
we  had  legislation  that  came  out  last  week  as  a  result  of  one  of  our  in- 
vestigations.  This  involved  a  study  of  the  purchase  of  jet  fuel  by  the 
Department  of  Defense.   We  compared  their  costs  to  the  airlines  and  found 
huge  overpayments.   Previously  we  had  done  an  investigation  on  the  cut-off 
of  oil  to  our  military,  especially  the  6th  fleet,  after  the  1973  Arab- 
Israeli  war.   Senator  Jackson,  Chairman  of  the  Subcommittee  ,  introduced  legis- 
lation providing  for  criminal  penalties  in  the  event  any  individual  or  corpora- 
tion willfully  cuts-off  oil  to  the  military.   But  Marilyn  is  correct  when 
she  says  that  our  responsibility  is  not  that  of  developing  legislation, 
holding  subcommittee  hearings  on  legislation  and  moving  it  to  the  full  com- 
mittee on  a  regular  basis.   We  hope  that  our  investigations  result  in  appro- 
priate remedial  legislation.   Two  weeks  ago,  for  example,  Senator  Jackson 
introduced  an  amendment  to  Senator  Williams'  securities  bill.   That  amend- 
ment gave  the  SEC  the  authority  to  require  the  validation  of  securities  and 
the  authentication  of  securities.   This  was  based  on  a  three  year  investiga- 
tion on  stolen  securities,  organized  crime,  movement  through  brokerage 
houses,  banks  etc.   The  bill  was  passed  by  the  Senate  and  is  now  in  con- 
ference.  We  consider  it  a  great  accomplishment  after  several  years  of  in- 
vestigation. 

The  Permanent  Subcommittee  on  Investigation  has  a  long  and  glorious 
history.   It  previously  was  known  as  the  McClellan  Committee  and  during 
that  time  we  went  through  labor  racketeering,  the  Valachi  hearings,  TFX 
and  others.   Prior  to  that  it  was  the  McCarthy  Committee,  and  I  have  great 
reservations  about  some  of  the  techniques  and  procedures  used  at  that  time. 
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But  as  a  result  of  the  McCarthy  days  we  have  very  rigid  procedures  and 
guidelines  governing  our  hearings  —  all  of  which,  by  the  way,  are  held 
under  oath  —  about  rights  to  counsel,  about  subpoenaing  documents  and 
other  safeguards.   We  run  what  I  term  investigative  rather  than  oversight 
hearings.   Seldom  do  we  have  anyone  before  our  Subcommittee  who  has  not  been 
interviewed,  whose  records  we  have  not  supoenaed  or  from  whom  we  have  not 
received  some  corroborating  information  about  the  investigation.   It  takes 
many  months  to  put  on  some  of  our  most  extensive  hearings,  although  we  have 
put  on  less  extensive  sessions  also. 

We  deal  quite  regularly  with  people's  lives  and  reputations,  and  I 
find  it  a  very  heavy  responsibility.   For  example,  when  we  put  on  our  se- 
curities hearings,  some  witnesses  would  mention  a  hundred  names  who  were 
involved  in  illegal  transactions.   We  had  to  go  out  and  try  to  run  down 
those  people  in  the  event  their  names  were  going  to  be  mentioned.   It  is 
tedious  work.   We  have  a  large  staff,  and  some  talented  veteran  investiga- 
tors.  That's  another  question.   How  do  you  pick  a  good  investigator?   It's 
very,  very  difficult.   They  can  be  ex-FBI  people,  they  can  be  newsmen,  they 
can  be  attorneys  or  others.   The  prerequisite  is  to  have  a  knack  to  ferret 
out  the  facts. 

We've  done  such  things  as  the  Russian  grain  deal,  cut-off  of  oil  to 
the  military,  the  CHAMPUS  investigations  (that's  the  military  spending  for 
psychiatric  care  of  benefits  for  children),  various  other  oil  investigations, 
the  Russian  grain  deal  II  (as  we  term  it  -  the  October  sale),  stolen  se- 
curities cases,  material  shortages  hearings  that  were  more  of  an  oversight 
nature  with  Senator  Huddleston,  when  he  was  on  our  subcommittee  handled 
those.   At  present,  we  are  involved  in  an  investigation  of  DEA  (the  Drug 
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Enforcement  Administration),  focusing  on  their  efficiency  and  operations. 
Our  jurisdiction  is  very  broad  -  and  I  should  point  here  that  our  Subcom- 
mittee is  in  turn  a  combination  of  two  subcommittees  which  were  combined 
in  March  of  1973:   the  Permanent  Subcommittee  on  Investigations  and  the 
old  National  Security  Subcommittee,  which  was  chaired  by  Senator  Jackson. 
That  Subcommittee  has  looked  into  many  national  security  questions  with  a 
very  talented  group  of  people  who  are  leading  experts  on  foreign  policy. 
Thus,  we  have  a  dual  jurisdiction  -  domestic  and  foreign.   The  resolution  is 
extremely  broad  and  it  permits  us  to  function  in  just  about  any  area.   We 
have  cultivated  relationships  with  state  and  local  police  and  with  other 
governmental  agencies.   Our  files  have  become  a  repository  for  many  scholars 
who  want  to  write  books  on  racketeering  and  organized  crime  and  things  of 
that  nature.   We've  tried  to  maintain  a  more  balanced  investigative  ap- 
proach, and  I  think  it 's  exhibited  in  the  kinds  of  things  we  are  doing  now. 
I  could  go  on  and  on  regarding  our  investigations,  but  I  don't  want  to  take 
up  your  time.   We  coordinate  with  all  the  other  subcommittees.   We've  been 
looking  at  utility  questions,  for  example,  and  Vic  is  the  expert  here.  We 
hope  to  build  on  things  that  he's  done.   So  with  that  I  will  depart. 

Does  anyone  have  any  questions  for  Mr.  Feldman  concerning  his  Subcommit- 
tee? Next  is  Mr.  Jim  Davidson,  Coune«l  to  Intergovernmental  Relations  Sub- 
committee, which  is  chaired  by  Senator  Muskie. 

Mr.  Jim  Davidson 

As  Marilyn  explained  the  Intergovernmental  Relations  Subcommittee 
has  principal  responsibility  for  the  relationships  between  the  Federal, 
State  and  local  governments,  funding  assistance  programs,  to  the  extent 
that  it  does  not  traditionally  fall  within  Judiciary  Committee  concerns 
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for  Federalism  and  constitutional  problems  of  proper  Federal  and  state 
roles  for  administering  or  financing  programs.   Finally  we  explore  general 
legislative  problems  where  Federal  legislation  has  an  import  on  the  opera- 
tion of  state  and  local  governments  apart  from  financial  problems. 

In  addition,  our  Subcommittee  ,  through  mutual  agreement  between  the 
Chairman  of  the  Judiciary  and  the  Government  Operations  Committees,  has 
inherited  a  certain  role  for  examining  government  policies  with  respect  to 
information  in  the  Freedom  of  Information  Act,  the  Privacy  Act,  and  the  rights 
of  members  of  Congress  and  congressional  committees  to  information  in  the 
possession  of  the  Executive  Branch  which  in  the  last  Congress  produced  the 
so-called  right  to  information  or  executive  privilege  bill  which  was  passed 
by  the  Senate  but  was  not  adopted  by  the  House.   I  think  this  will  give  you 
an  idea  of  the  types  of  oversight  roles  that  we  have  fulfilled  during  the 
last  Congress.   We  did  extensive  hearings  and  investigations  of  the  General 
Revenue  Sharing  program  in  June  of  last  year  when  we  held  four-days  of 
hearings  oversight  on  the  implementation  of  the  program.   We  are  attempting 
to  examine  the  limited  prescriptions  on  the  use  of  revenue  sharing  funds, 
how  the  funds  were  being  used,  whether  or  not  they  were  being  used  in 
violation  of  any  provision  of  the  Revenue  Sharing  Act  with  emphasis  on 
the  civil  rights  provisions  of  that  Act.   Last  years  hearings  involved 
the  city  of  Chicago  and  its  use  of  revenue  sharing  funds  in  support  of  the 
police  department  work  with  focus  of  attention  there.   Those  hearings  and 
ones  yet  to  come  will  figure  strongly  in  this  year's  congressional  con- 
sideration of  the  renewal  of  revenue  sharing. 

Also  last  year  we  focused  on  the  impact  of  the  energy  crisis  on  state 
and  local  governments.   This  was  a  financial  as  well  as  an  administrative 
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problem,  looking  at  the  cost  of  state  and  local  governments  of  dealing  with 
shortages  of  energy  as  well  as  the  impact  on  constituents  of  those  jurisdic- 
tions and  the  problems  they  were  facing  with  State  and  local  administration 
of  Federal  energy  programs.   That  investigation  required  an  extensive  sur- 
vey of  all  states  and  their  energy  offices  and  how  they  were  implementing 
Federal  and  state  energy  programs.   That  resulted  in  a  staff  study  that  was 
published  in  January  of  this  year.   The  study  also  drew  from  hearings  that 
were  held  both  last  June  and  last  October  on  the  Federal  energy  crisis  and 
its  impact  on  state  and  local  governments. 

As  I  also  mentioned  we  have  tried  to  examine  the  relationship  and  the 
impact  of  the  concept  of  new  Federalism  when  this  was  a  central  theme  of  the 
Nixon  Administration  —  that  is  how  State  and  local  governments  were  attempting 
to  deal  with  the  decentralization  of  the  administration  of  Federal  programs 
as  greater  emphasis  was  being  placed  on  the  regional  offices.   That  included 
the  questions  of  how  this  reorientation  was  actually  working  -  was  the  shift 
of  responsibility  going  out  to  the  regional  off ices, or  are  they  simply  still 
figureheads; and  was  in  fact  the  decision  for  state  and  local  programs  still 
being  made  in  Washington?  We  had  eight  days  of  hearings  on  the  new  Federalism 
program  bringing  in  the  secretaries  of  HEW,  HUD  and  the  Director  of  the  Of- 
fice of  Management  and  Budget. 

The  Subcommittee,  also  ir 
level  of  public  confidence  and  concern  about  Federal  Government,  State  govern- 
ments, and  local  governments  and  the  leadership  and  administration  of  programs 
at  each  of  those  levels  of  government.   The  Subcommittee  retained  the  Louis 
Harris  organization  to  conduct  a  poll  of  American  citizens  and  also  more  than 
206  local  and  state  officials  were  interviewed  during  this  process  -  both 
in  person  and  through  written  questionnaires  attempting  to  evaluate  what  the 
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public" 8  perception  was  of  the  administration  of  local,  State  and  Federal 
programs  and  in  turn  what  officials  at  those  levels  of  government  thought 
the  public's  perception  was  of  the  administration  of  government  programs. 
That  produced  an  interesting  split  in  the  evaluation.   There  was  a  much 
greater  feeling  that  a  better  job  was  being  done  by  those  holding  office 
than  by  those  who  elected  them.   That  study  was  published  in  December  of 
1973  in  3-volumes  and  still  used  as  a  basis  of  evaluating  public  perceptions 
of  local,  State  and  Federal  programs. 

In  the  area  of  constitutionally  related  questions  of  the  Congressional 
right  to  information  as  well  as  freedom  of  information  and  privacy,  our 
Subcommittee  played  a  principal  role  in  developing  certain  amendments  to 
the  Freedom  of  Information  Act  in  conjunction  with  the  Judiciary  Committee. 
Also  we  conducted  with  the  Judiciary  Committee  a  series  of  hearings  in  1973 
and  followed-up  with  hearings  last  year  into  questions  of  government  se- 
crecy, the  withholding  of  information  both  from  the  public  and  from  the 
Congress  and  the  related  constitutional  questions  involved  as  I  mentioned 
before,  that  produced  the  so-called  Congressional  Right  to  Information 
Act  which  was  enacted  by  the  Senate  and  which  will  be  reintroduced  again 
by  our  Subcommittee  this  year.   In  the  area  of  government  secrecy  - 
we  held  in  1973  a  total  of  7  days  of  hearings.  Last  year  we  conducted 
5  days  of  hearings  into  the  Executive  Branch  administration  of  Executive 
Order  11652  which  was  the  most  recent  in  a  series  of  Presidential  Execu- 
tive Orders  prescribing  the  procedures  for  classifying  government  secrets 
confidential,  secret  and  top  secret.  We  hope  that  another  round  of 
hearings  this  year  will  help  produce  Congressional  guidelines  for  the 
classification  of  government  secrets  which  will  replace  the  Executive 
Order  guidelines  for  classification. 
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The  Subcommittee  through  the  full  committee  played  a  major  role  in 
the  development  of  the  Privacy  Act  of  1974  and  this  year  is  working  toward 
the  implementation  of  that  Act  which  goes  into  effect  September  27th.  We 
are  working  almost  on  a  daily  basis  with  the  Office  of  Management  and 
Budget  in  the  development  of  guidelines  for  the  implementation  of  the  Act, 
and  plan  hearings  in  the  very  near  future  on  oversight  of  the  level  of 
preparedness  of  various  agencies  toward  implementing  the  Privacy  Act. 

One  incidental  role  of  the  Subcommittee  which  will  probably  be  con- 
tinued in  this  Congress  involved  hearings  on  the  Congressional  oversight 
of  U.S.  intelligence  agencies.   In  December  of  last  year  we  had  two  days 
of  hearings  and  had  planned  follow-up  hearings  in  January  and  February 
until  the  N.Y.  Times  broke  the  story  of  intelligence  agency  surveillance 
of  U.S.  citizens.   It  was  our  Subcommittee  that  developed  the  resolution 
which  the  Senate  adopted  establishing  a  Senate  Select  Committee  on  In- 
telligence Oversight  and  will  probably  receive  again  the  referral  of  any 
recommendation  by  that  Select  Committee  for  the  establishment  of  a  per- 
manent Oversight  Committee  on  Intelligence.  As  you  can  see,  our  Sub- 
committee has  rather  diverse  responsibilities,  but  part  of  it  is  by 
tradition  and  part  of  it  is  just  by  agreement  between  the  Chairmen  of 
the  Government  Operations  Committee  and  Judiciary.   I  will  defer  to 
others  if  you  have  any  questions. 

Thank  you.   Bill  Goodwin  is  staff  director  of  the  Oversight  Sub- 
committee, the  new  Subcommittee  which  I  mentioned  previously.  As  you 
may  know,  the  proceedings  of  this  conference  or  seminar  are  going  to  be 
published  in  a  document  which  his  Subcommittee  will  be  preparing. 
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Mr.  Bill  Goodwin 

Well  very  briefly,  our  Subcommittee  was  created  at  the  first  of  this 
year. 

We  have  four  primary  duties.   First,  the  development  of  better  methods 
and  procedures  of  congressional  oversight.   Second,  the  study  of  the  utili- 
zation and  disposal  of  Federal  property.   Third,  the  consideration  of  budget 
measures  other  than  those  within  the  jurisdiction  of  the  Budget  Committee. 
Fourth,  general  oversight  of  the  work  of  the  Federal  Paperwork  Commission, 
which  was  created  at  the  end  of  last  year.   Of  these  our  work  in  oversight 
is  our  major  responsibility. 

One  of  the  chief  reasons  that  our  Subcommittee  was  created  was  the 
general  feeling  that  Congress  can  do  a  better  job  of  oversight.   As 
Walter  said  at  the  beginning  of  this  seminar,  the  climate  is  right  for  im- 
proving our  oversight  functions.   You  might  say  that  the  question  is 
whether  we  are  going  to  have  oversight  by  scandal  —  do  we  have  to  create 
a  select  committee  every  time  we  have  a  Watergate  or  a  revelation  in  the 
N.Y.  Times  about  C.I. A.  activities. 

The  House,  I  think,  got  the  jump  on  the  Senate  with  its  committee 
reforms  and  the  House  Government  Operations  Committee  now  has  new 
strength  and  responsibilities  in  the  oversight  area. 

Our  Sttbcommittee  is  planning  to  survey  the  Senate  committees, 
which  will  be  similar  to  the  recent  report  of  the  House  Government  Opera- 
tions Committee  —  although  we  do  not  have  the  authority  by  Senate  rule 
that  the  House  committee  has  by  House  rule  at  the  present  time. 

We  also  intend  to  put  together  in  committee  print  form  a  fairly  compre- 
hensive reference  work  on  oversight  sometime  within  the  next  two  months. 
This  will  be  a  document,  which  we  intend  to  make  available  to  all  members 
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and  all  committees  giving  outlining  the  different  types  and  different 
techniques  of  oversight.   In  effect,  we  will  take  this  seminar  and  ex- 
pand it  into  more  technical  avenues. 

We  also  intend  to  initiate  studies  of  our  own  in  oversight,  and 
to  use  the  resources  of  the  General  Accounting  Office  and  of  CRS.   We  hope 
to  be  able  to  serve  as  a  coordinator,  and  to  more  or  less  advertise  the 
work,  that  is  going  on  in  CRS  and  in  GAO. 

Our  primary  function  is  to  work  on  better  committee  functions  of  the 
oversight  process  and  to  come  up  with  ideas  of  how  the  Senate  committees 
can  better  oversee  the  operations  of  the  executive  branch  on  a  day-to-day 
and  month-to-month  basis  as  opppsed  to  waiting  for  some  scandal  to  break 
in  the  Washington  Post.   It's  not  going  to  be  easy,  but  budget  reform 
wasn't  easy,  either.   As  I  recall,  Vic,  it  took  2  1/2  to  3  years  to  get 
that  bill  through  the  Congress  -  so  we  don't  visualize  results  overnight. 

In  the  meantime,  our  Subcommittee  will  be  doing  some  oversight  of  its 
own,  especially  of  GSA  as  it  pertains  to  the  use  and  disposal  of  Federal 
property  —  not  the  procurement  of  property.   The  Subcommittee  recently 
staffed  oversight  hearings  in  the  full  Government  Operations  Committee  on 
energy  conservation  to  try  to  get  a  grasp  of  what  all  the  agencies  are 
doing  in  energy  conservation  and  what  sort  of  coordination  is  taking 
place  in  the  Executive  Branch.   We  also  will  be  watching  the  Federal 
Paperwork  Commission  and  the  National  Commission  on  Productivity  and 
Work  Quality  which  should  be  reformed  within  the  next  2  or  3  months  into 
a  National  Center  for  Productivity. 

Those  are  some  of  the  Subcommittee's  own  oversight  plans. 

The  staff  hopes  to  be  of  help  to  everybody,  frankly.  We  are  not 
going  dabbling  into  other  people's  jurisdiction.   We  want  to  put  together 
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information  and  ready  references  that  will  be  helpful  to  all  members  of 
the  Congress  and  their  staffs  and  all  the  committees  and  their  staffs 
and  to  proceed  with  the  development  of  better  methods  by  which  Congress 
can  do  its  oversight  job. 
Ms.  Marilyn  Harris 

Next,  Mr.  Vic  Reinemer,  the  staff  director  for  the  Reports,  Accounting, 
and  Management  Subcommittee. 
Mr.  Vic  Reinemer 

I  see  the  House  Government  Operations  Committee  staff  came  in  so 
please  impose  their  five  minute  fule  on  me  if  I  go  beyond,  so  that  they 
have  plenty  of  time. 

I  would  like  to  just  mention  just  two  of  our  oversight  activities 
because  they  involve  some  different  techniques  which  may  be  of  interest 
to  you.   They  also  both  afford  examples  of  something  which  we  always  find 
in  evaluation  which  makes  our  job  more  difficult,  which  I  think  we  ought 
to  talk  about  briefly.   And  that's  the  congressional  compulsion  to  over- 
legislate.   It's  part  of  the  process.   The  members  see  a  situation  that 
needs  correcting.   They  say  there  ought  to  be  a  law,  let's  put  out  a  press 
release,  and  put  it  on  through.  And  then  we  get  really  this  extraordinary 
confusion  of  existing  laws,  sometimes  conflicting  laws.   So  when  we  do  try 
to  evaluate  or  make  a  particular  program  work,  instead  of  going  back  and 
using  a  much  better  law  —  and  here  the  73rd  and  74th  Congresses  offer 
some  beautiful  precedents  —  instead  we  try  to  legislate  some  more. 

This  is  what  is  going  on  right  now  in  the  energy  field,  especially  re: 
foreign  ownership  of  energy  corporations  and  other  corporations.   Instead 
of  working  with  the  various  regulatory  agencies,  which  have  all  sorts  of 
authority  to  collect  and  tabulate  and  make  meaningful  reports  on  ownership 
of  companies  foreign  or  otherwise,  we  are  passing  more  laws,  and  not  paying 
as  much  attention  as  we  all  should  in  my  opinion  to  the  development  of  the 
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regulations  and  the  enforcement  of  the  various  reporting  require- 
ments. 

I  was  so  glad  to  hear  Ellen  Collier  earlier  in  the  day  talk  about 
the  importance  of  the  agency  reports.   But  what  I  am  talking  about  here 
is  the  corporate  reports  to  the  regulatory  commissions,  and  the  information 
within  them,  which  is  so  important  to  enforcement  of  laws  whether  it  be 
anti-trust,  conflict  of  interest  or  a  rule-making  proceeding  or  whatever. 
So  I  just  wanted  to  make  this  point  —  that  maybe  we  will  have  more  peo- 
ple like  Senator  Barkley,  who  made  quite  a  point  of  not  introducing  legis- 
lation —  except  occassionally  something  having  to  do  just  with  Kentucky. 
He  prided  himself  on  being  a  developer  of  particular  refinements  of  legis- 
lation rather  than  being  the  first  to  put  his  name  on  the  bill. 

Well,  anyway,  when  we  dealt  with  this  matter  of  corporate  disclosure 
our  technique  was  first  the  traditional  study,  in  which  we  obtained  the 
cooperation  of  CRS  and  GAO,  then  came  the  hearings  at  which  it  became  a- 
bundantly  clear  that  the  various  regulatory  commissions  had  all  the 
powers  they  needed  to  cc'lect  information  as  to  who  owned  and  controlled 
various  companies  that  report  to  the  regulatory  commission.  We  also 
saw  that  the  regulations  had  just  grown  up  topsy  turvy. 

So  as  a  result  of  these  hearings  where  we  established  that  we  didn't 
need  laws  we  —  and  here  the  GAO  was  instrumental  —  established  a  steering 
committee  on  uniform  corporate  reporting  requirements.   And  now  when  they 
go  out  for  rule-making  comes  the  real  follow-through.   For  example  the  ICC 
has  formally  proposed  adoption  of  these  model  corporate  reporting  re- 
quirements that  deal  with  the  stock-holdings,  debt-holdings,  corporate 
structure  and  affiliations  of  officers  and  directors,  of  major  regulated 
companies.   If  the  ICC  and  the  other  commissions  adopt  these  requirements 
we  are  going  to  have,  all  of  us,  for  the  first  time,  a  great  deal  of 
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current  information  which  has  otherwise  been  obtainable  partially  only 
by  special  and  time-consuming  proceedings.   But  the  difficulty  here  is 
that  what  adoption  of  the  model  disclosure  regulations  requires  is  for 
various  members  and  interest  groups  to  actually  become  parties  in  ad- 
ministrative proceedings.   So  this  is  one  thing  that  Senator  Metcalf  and 
a  few  others,  especially  on  the  House  side  too,  are  increasingly  doing  — 
being  formal  parties  to  administrative  proceedings  of  the  regulatory 
commissions.   The  other  activity  that  I  want  to  mention  briefly  has  to 
do  with  the  advisory  committees.  This  Federal  Advisory  Committee  Act  is 
two  years  old  now,  two  years  and  4  months.   And  actually  the  number  of 
Federal  advisory  committees  has  decreased  from  1439  to  —  OMB  and  GSA 
have  a  different  count,  but  Jerry  Sturgis  here  has  the  best  count, 

what  is  your  figure  Jerry 1242.   So  we  have  gotten  rid  of  about  200. 

And  that  is  a  net  reduction.  Last  year  Congress  established  53  new 
Federal  advisory  committees,  by  statute,  despite  the  very  explicit 
language  in  our  two-year-old  statute  that  no  new  Congressional  statutory 
advisory  committees  shall  be  established  unless  there  is  proven  that 
there  isn't  another  one  that  can  do  the  job  or  perhaps  the  agency  it- 
self could  do  the  job. 

We  really  are  looking  for  some  help  on  how  to  handle  this  one.  What 
we  did  about  a  year  and  a  half  ago  was  to  send  to  all  the  standing  com- 
mittees a  list  of  all  the  advisory  committees  which  are  subject  to  their 
jurisdiction.  Now  we  are  going  to  send  out  a  little  stronger  document 
this  time.   It  will  include  a  list  of  all  the  new  advisory  committees 
that  they  established,  probably  not  in  accordance  with  the  law  which 
had  been  resoundingly  approved.  Here  is  the  sort  of  an  area  where  we 
are  going  to  continue  in  and  to  ask  the  CRS  for  some  assistance.   It 
has  been  so  helpful  on  indexing.   We  will  soon  have  available  the  1975 
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report  on  all  the  advisory  committees.   This  will  include  the  membership, 
meeting  information  and  all  that.   That's  already  available  on  microfilm. 
And  by  Labor  Day  we  will  have  an  alphabetized  index,  as  we  did  after  the 
first  report  came  out  two  years  ago.   This  will  be  alphabetized  both  as  to 
institution  and  individual.   It  will  be  a  very  valuable  resource. 

Ms.  Marilyn  Harris 

Thank  you,  Vic.   I  would  like  Ron  Chiodo  to  share  with  you  the  re- 
sponsibility of  the  Federal  spending  practices,  Efficiency  and  Open  Govern- 
ment Subcommittee,  chaired  by  Senator  Chiles. 

Mr.  Ron  Chiodo 

Ours  also  is  a  relatively  new  Subcommittee.   In  1969  by  law  the 
Commission  on  Government  Procurement  was  formed  to  review  the  procurement 
activities  of  the  Federal  Government  and  recommend  any  necessary  reforms. 
In  1972  their  study  was  completed  and  included  149  recommendations.   As 
a  result  the  Ad  Hoc  Subcommittee  on  Procurement  was  established  in  the 
Government  Operations  Committee  last  year  to  carry  out  the  Commission's 
recommendations.   This  year  the  jurisdiction  has  been  broadened,  and  now 
includes  Federal  spending  practices,  efficiency  and  open  government. 

Very  briefly,  the  Federal  spending  practices  area  includes  Federal 
procurement  and  the  laws,  regulations  and  procedures  governing  Federal 
contracts,  grants,  transfer  payments,  and  other  spending  arrangements 
includes  the  Office  of  Federal  Procurement  Policy,  which  was  established 
just  this  year.   It  also  includes  all  Executive  Branch  organizations 
which  are  responsible  for  Federal  spending  practices.   To  the  extent  that 
"Federal  spending  practices"  doesn't  cover  the  ball  park,  we  have 
"efficiency"  which  includes  the  efficiency  and  economy  of  Federal  spending 
practices  generally.   Finally,  of  course,  we  have  "open  government"  and 
that  particularly  has  to  do  with  legislation  regarding  the  public  con- 
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duct  of  meetings  of  all  branches  of  the  Government. 

Just  to  give  you  a  few  examples,  in  the  Federal  Spending  Practices 
area  we  are  in  the  process  of  preparing  for  and  will  be  conducting  over- 
sight hearings  with  regard  to  how  the  Federal  Government  goes  about 
procuring  major  systems,  be  they  weapons  systems  or  civil  systems.   The 
Department  of  Defense  has  procurement  programs  today  which  will  total 
about  $140  billion  when  acquisition  is  complete.   The  dollar  figures 
involved  are  very  high  and  the  cost  overruns  that  we've  all  noted  in  the 
newspapers  contributes  significantly  to  this.   There  is  some  concern  that 
there's  a  lack  of  competition  in  the  acquisition  of  major  programs.   We 
also  see  the  problems  of  interservice  rivalry  that  have  to  be  contended 
with.   In  the  civil  area,  GAO  has  done  a  recent  report  on  civil  acqui- 
sitions and  has  found  some  projects  whose  cost  overruns  have  been  as  high  as 
200%.   So  not  all  the  problems  are  in  the  area  of  the  Department  of  Defense. 
Continuing  in  this  one  area  of  oversight ,  we  also  have  concern  over  the 
way  the  Federal  Government  acquires  all  kinds  of  property,  be  it  major 
systems  or  perhaps  just  desks  and  typewriters. 

Today  the  acquisition  process  for  the  government  involves  something 
over  4,000  laws  that  are  sometimes  conflicting  -  certainly  not  all  in 
one  place.  There  is  a  need  to  revise  and  streamline  the  acquisition 
process  generally.  We  are  now  working  on  legislation  which  would  do 
that  -  bring  the  procurement  laws  together  in  a  consolidated  acquisition 
statute  and  amend  the  existing  laws  as  is  necessary  to  develop  a  con- 
sistent approach  to  the  acquisition  of  property. 

By  calling  ourselves  Subcommittee  on  Federal  Spending  Practices 
rather  than  just  Procurement  as  was  the  ease  last  year,  we  also  have  clear 
jurisdiction  over  the  grants  area  and  we  are  conducting  investigations 
with  regard  to  grant  programs,  HEW  comes  to  mind  here.   The  dollar  figures 
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involved  again  are  very  high,  upward  of  $50  billion.  We  see  the  same 
thing  I  think  you  all  have  noted,  that  in  some  cases  the  process  by 
which  grants  are  let  can  be  considered  very  confused.   There  have  been 
many  examples  in  which  very  little  discipline  has  been  applied  to  the 
system.   There  have  been  some  isolated  examples  where  awards  have  cer- 
tainly wired  to  favored  recipients,  perhaps  suggesting  fraud.  We  have 
introduced  some  legislation  with  regard  to  the  grants  area  to  help 
clarify  the  circumstances  under  which  a  grant  should  apply  and  the  cir- 
cumstances under  which  instead  a  contract  should  apply. 

With  regard  to  the  second  area  of  jurisdiction,  that  of  the  effi- 
ciency of  Federal  programs,  we  have  been  looking  at  the  social  security 
program  as  one  example,  and  the  food  stamp  program  as  another  example. 
Questions  that  are  being  raised  are  primarily  with  regard  to  the  mechanism 
that's  being  employed.   Are  applicants  getting  a  fair  shake  out  of  the 
system?  Is  the  system  too  overloaded  with  bureaucracy?  Is  there  a 
better  mechanism  for  providing  the  same  benefits?   In  no  case  does 
our  jurisdiction,  though,  extend  to  the  objectives  whether  we  should 
have  a  food  stamp  program  or  social  security  program.  We  would  only 
raise  questions  as  to  whether  the  program  was  being  conducted  as  effi- 
ciently as  possible. 

The  last  area  of  jurisdiction  of  open  government  is  rather  obvious. 
There  is  legislation  which  has  been  introduced  this  year  called  Sun- 
shine legislation.   (Audience:  What  state  originated  the  basic  legis- 
lation?) Ron:   State  of  Florida.   Florida  was  one  of  the  first  states 
that  had  pursued  legislation  to  open  its  government  meetings  to  the  public 
and  the  term  "sunshine"  has  been  applied  to  legislation  of  this  type. 
S.  5,  a  sunshine  bill,  is  tentatively  scheduled  for  mark-up  this  month. 
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This  briefly  covers  our  area  of  legislative  oversight.  We  also  have 
the  typical  activities,  case  work  load  and  investigations  that  come  up 
when  particular  violations  of  law  are  found  just  like  many  of  the  other 
subcommittees  do.   I  will  turn  it  back  to  Marilyn  if  there  are  any  ques- 
tions. Marilyn:  Are  there  any  overall  questions  on  the  Senate  side? 

Audience 

Just  one,  Mr.  Goodwin  didn't  say  who  the  members  of  his  committee  are 
and  I  would  be  interested 

Bill 

The  Chairman  is  Senator  Nunn.  Other  majority  members  are  Senators 
Jackson,  Muskie,  Allen,  and  Chiles.   The  ranking  minority  member  is 
Senator  Percy  and  Senator  Roth. 
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DAY  II:   CONGRESSIONAL  COMMITTEE  CASE  STUDIES  OF  EVALUATION 
AND  OVERSIGHT:   SUMMARY  OF  PROCEEDINGS 


Committee  staff  experiences  in  doing  or  using  various  types  of 

evaluation,  quantitative  evaluation  research,  and  oversight  formed  the 

topic  for  the  second  day  of  the  seminar,  May  12. 

A.   Oversight  and  Evaluation  in  the  House  Committee  on  International 
Relations:  On-site  Staff  Visits 

The  proceedings  commenced  with  a  presentation  hy  Jack  Sullivan,  staff 
Consultant  to  the  House  Committee  on  International  Relations  (formerly 
the  House  Committee  on  Foreign  Affairs).   Dr.  Sullivan  began  his  presentation 
by  outlining  the  history  of  oversight  activities  in  the  International 
Relations  Committee.   Then  he  described  the  Committee's  current  sub- 
committee structure.   In  response  to  H.  Res.  988,  the  Committee  Reform 
Amendments  of  1974,  the  Committee  was  given  new  jurisdictional  responsibili- 
ties.  It  created  three  new  oversight  Subcommittees  to  deal  with  existing  and 
new  responsibilities.   These  are:   (1)   the  Subcommittee  on  Oversight,  which 
deals  with  oversight  of  foreign  intelligence,  administration  of  foreign  affairs 
external  research,  and  all  areas  not  covered  by  other  subcommittees;  (2)   the 
Subcommittee  on  Investigations,  which  deals  with  special  foreign- policy- re- 
lated investigations  referred  by  the  chairman  with  the  consent  of  the  com- 
mittee; and  (3)   the  Subcommittee  on  Future  Foreign  Policy  Research  and 
Development. 

Most  of  Dr.  Sullivan's  discussion  focused  on  how  the  Committee  and 
Subcommittees  have  gathered  information  for  oversight  of  overseas  operations 
of  the  U.S.  Government.   The  Committee  uses  several  techniques,  including 
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onaite  visits  and  related  analysis  by  neutral  bodies,  such  as  the  CRS 
or  the  CAO,  and  visits  by  members  of  Congress  and  committee  staff  investiga- 
tors.  Dr.  Sullivan  emphasized  the  importance  to  both  the  International' 
Relations  and  the  Foreign  Relations  Committees  of  onsite  overseas  investi- 
gative staff  visits. 

He  gave  several  case  study  examples  to  illustrate  the  utility  of  such 
visits  as"  an  evaluation  and  oversight  information  gathering  technique.   For 
example,  while  making  an  onsite  visit  in  Asia,  Committee  staff  investiga- 
tors who  interviewed  local  AID  recipients  learned  that  many  women  in  Indonesia 
and  the  Phillipines  using  AID-supplied' birth  control  pills  were  suffering  a 
variety  of  side-effects.   AID  headquarters  and  the  chief  AID  physician 
discounted  the  importance  and  severity  of  these  effects.   They  were  in- 
terested in  evaluating  the  success  of  the  AID  birth  control  distribution 
program  in  terms  of  numbers  of  persons  participating  (a  process  evaluation 
question) ,  rather  than  also  in  terms  of  the  impacts  of  the  program  on  the 
participants.   After  Committee  staff  reported  back,  to  the  full  Committee, 
congressional  pressure  forced  local  AID  administrators  to  investigate  and 
confirm  that  women  were  indeed  suffering  side-effects.   Additional  onsite 
investigation  indicated  that  due  to  GSA  procurement  regulations,  each  year, 
on  the  basis  of  lowest  bid,  a  different  pharmaceutical  firm  supplied  AID 
with  birth  control  pills.   The  chemical  composition  of  the  pills  varied 
significantly  enough  to  casue  side-effects  in  women  who  were  given  new 
types  of  pills  every  fiscal  year.   Finally,  congressional  pressure  and 
publicity  caused  the  AID  physician  to  change  his  position  and  the  GSA 
to  change  procurement  regulations  so  that  women  could  receive  the  same 
drug  continuously. 
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Dr.  Sullivan  indicated  that  onsite  visits  are  only  one  important 
form  of  information-gathering  for  evaluation  and  oversight.   However, 
he  stressed  that  they  are  especially  useful  because  they  generate 
information  that  cannot  be  obtained  by  any  other  method  including  the 
use  of  neutral  research  bodies.   Onsite  committee  staff  investigators, 
unlike  those  of  neutral  research  bodies,  have  immediate  access  to  de- 
cisionmakers: that  is,  members  of  Congress.   It  is  important,  Dr.  Sullivan 
stated,  for  investigators  to  have  daily  close  contact,  trust,  and  rapport 
with  members.   This  relationship  aids  immeasurably  in  providing  access  for 
using  evaluation  and  oversight  information  in  policymaking. 

Dr.  Sullivan  emphasized,  however,  that  two  important  criteria  must 
be  met  to  insure  the  success  of  onsite  overseas  staff  investigations: 
(1)  the  investigators  must  be  highly  trained  professionals,  and  (2)  visits 
should  be  made  in  pairs,  to  aid  the  committee  in  corroborating  evidence 
and  statements  made  in  interviews  which  Federal  agency  administrators 
might  deny  when  testifying  before  the  Congress. 

B.   Oversight  and  Evaluation  in  the  House  Committee  on  Government 
Operations:  The  Freedom  of  Information  Act 

William  G.  Phillips,  who  from  1971  to  1975  was  staff  director 

of  the  Foreign  Operations  and  Government  Information  Subcommittee  of 

the  House  Government  Operations  Committee,  gave  the  second  presentation 

of  the  afternoon. 
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Mr.  Phillips  began  his  statement  by  providing  a  detailed  descrip- 
tion of  the  sequence  of  oversight  and  investigative  activities  and 
then  legislative  action  which  led  to  congressional  passage  and 
overriding  of  a  presidential  veto  of  amendments  to  the  Freedom  of  In- 
formation Act  of  1966,  P.L.  89-487.   The  act  provides,  in  general, 
that  '"any  person'  might  have  access  (with  certain  exceptions)  to 
rules,  opinions,  orders,  and  other  types  of  public  records  of  various 
agencies  of  the  Executive  branch  of  the  Federal  government." 

Mr.  Phillips  noted  how  initial  oversight  was  hampered  since  the  more 
than  a   hundred  affected  agencies  interpreted  the  Act's  access  and 
reporting  responsibilities  differently.  A  committee  print  describing 
these  and  other  difficulties  was  published.   The  Committee  then  expanded 
its  oversight  by  administering  a  detailed  questionnaire  to  agencies 
to  determine  the  ways  in  which  the  agencies  complied  with  the  law 
during  the  period  1967  to  1971.   After  obtaining  assistance  from  the 
CRS  in  analyzing  the  responses,  the  Committee  held  a  set  of  hearings 
and  then  prepared  a  report  on  problems  with  administration  of  the  act. 

Recommendations  for  improvement  were  included  in  bipartisan  amending 
bills  introduced  in  the  House.   Simultaneous  oversight  hearings  were 
conducted  in  the  Senate,  followed  by  House  and  Senate  passage  of  a 
compromise  version  of  a  bill  to  strengthen  the  Act.   The  Administration 
opposed  the  bill  and  the  President  vetoed  the  act.   However,  the  Congress 
overrode  the  veto  on  the  strength  of  oversight  information  generated  and 
because  the  climate  —  the  era  of  the  Watergate  scandals  —  provided 
enough  incentive  for  a  bipartisan  congressional  reaction. 
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Mr.  Phillips  made  two  additional  important  points  in  concluding  his 
remarks.   First,  he  made  a  clear  differentiation,  based  on  his  experiences, 
among    three  different  types  of  operational  congressional  oversight: 
(1)   "routine":   that  is,  monitoring  the  issuance  of  agency  regulations 
(published  in  the  Federal  Register)  to  see  if  the  agency  is  following 
the  intent  of  the  law  enacted  by  the  Congress;  (2)   "special":   in-depth, 
long-term  investigative  oversight  of  a  high-priority  area  which  might 
eventually  require  corrective  legislative  action,  for  instance  like  that 
done  on  the  amendments  to  the  Freedom  of  Information  Act;  and  (3)   "gueril- 
la":  spot  checking,  usually  in  an  adversary  or  conflict  situation  where 
a  set  of  one-day  hearings  prompts  corrective  action  by  a  Federal  agency 
without  legislative  enactment. 

Second,  he  noted  that  it  had  been  exceedingly  difficult  to  do  good 
oversight  before  passage  of  the  Committee  Reform  Amendments  of  1974, 
since  most  available  resources  had  to  be  used  to  prepare  new  legisla- 
tion.  As  a  result  the  Congress  had  to  rely  almost  exclusively  on  over- 
sight reports  prepared  by  Federal  agencies.   In  many  cases  these  reports 
were  biased  to  give  a  favorable  picture  of  a  sometimes  unsavory  situation. 
Mr.  Phillips  said  that  he  expects  the  House  committee  reorganization 
and  new  oversight  resources  and  responsibilities  to  aid  the  Congress 
significantly  in  ensuring  good  government  and  maintenance  of  the  balance 
of  powers. 

C.   Oversight  and  Evaluation  in  the  Joint  Economic  Committee:   The 
Family  Assistance  Plan 

For  three  and  one-half  years,  during  the  period  1971-1974,  the  Sub- 
committee on  Fiscal  Policy  of  the  Joint  Economic  Committee  conducted  a 
comprehensive  assessment  of  public  welfare,  resulting  in  a  24  volume 
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series  of  hearings,  reports  and  background  documents.   The  objective  of  the 
study  was  to  assess  the  effectiveness  and  impact  of  the  Family  Assistance 
Plan,  FAP,  which  established  a  Federal  minimum  income  floor  for  all  families 
with  children.   The  study  was  initiated  by  Representative  Martha  Griffiths, 
Cnairperson  of  the  Subcommittee  and  a  member  of  the  Ways  and  Means  Committee,  which 
had  oversight  responsibilities  for  the  program.   Ways  and  Means  Committee 
deliberations  on  the  issue  had  convinced  Mrs.  Griffiths  that  an  adequate 
review  of  FAP  required  that  the  program  be  assessed  in  the  context  of  all 
income  supplement  programs,  since  changes  intended  in  one  program  are  often 
counteracted  or  duplicated  by  other  income  benefit  programs.   The  other  sup- 
plemental benefit  programs  involved  were  those  offering  benefits  in  food, 
health  care,  day  care,  housing,  social  insurance,  social  security,  and  unem- 
ployment compensation. 

Dr.  Alair  Townsend,  research  director  for  the  Fiscal  Policy  Subcommittee 
which  conducted  the  study  series,  discussed  the  origins  of  the  study,  re- 
sources required  to  conduct  investigatory  evaluation  and  oversight  of  this 
nature,  methods  to  present  findings,  and  appropriate  types  of  products. 

All  the  income  benefit  programs  were  assessed  with  respect  to  (1)  the 
distribution  of  benefits,  especially  in  various  combinatons,  (2)  questions 
of  adequacy,  (3)  the  impact  on  incentives  with  respect  to  family  structure 
and  work  effort,  and  (4)   administrative  efficiency. 

Ms.  Townsend  summarized  several  prerequisites  for  conducting   a 
wide-ranging  and  lengthy  study  of  this  nature.   They  include:   (1)   adequate 
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leadtime  and  (2)   highly  trained  professional  committee  staff  cap- 
able of   close  teamwork.   This  is  necessary  to  permit  the  close  inter- 
facing required  between  Subcommittee  members  and  investigative  staff 
to  keep  the  study  on  target  and  to  enable  the  staff  to  monitor  closely 
the  work  of  the  internal  staff  and  outside  professionals.   Support 
agencies  which  assisted  in  the  research  included  the  Congressional  Re- 
search Service,  the  General  Accounting  Office,  the  House  Information 
Systems  Office,  and  outside  academic  researchers  working  on  contract  to 
the  Subcommittee. 

The  study  demonstrated  a  proliferation  and  fragmentation  of  pro- 
grams.  Responsibilities  for  implementation  and  oversight  were  frag- 
mented among  Federal  agencies  and  State  and  local  governments.   The 
research  also  identified  policy  problems,  inadequacies  in  data  collec- 
tion and  analysis,  and  the  need  for  better  syntheses  of  evaluation  re- 
search dealing  with  the9e  issues. 

The  Subcommittee  reports  consisted  of  24  volumes  of  background 
studies  and  four  sets  of  hearings  on  issues  raised.   These  detailed 
reports  were  distributed  widely  to  interested  committee  staff,  interested 
members,  Federal  agency  administrators,  and  academics.   However,  in  order 
to  achieve  wide  dissemination  of  results  among  members  of  Congress,  the 
reports  were  supplemented  by  nontechnical  summaries,  including  graphs 
and  charts . 

Ms.  Townsend  concluded  by  noting  that  although  the  Joint  Economic  Committee 
has  no  legislative  authority,  bills  containing  the  essence  of  the  findings 
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of  the  study  were  introduced  by  Mrs.  Griffiths  and  members  of  other 
committees  with  substantive  jurisdiction  for  issues  raised.   The  study 
received  wide  distribution  and,  in  Ms.  Townsend's  opinion,  has  had  a 
significant  impact  on  legislative  oversight  and  evaluation. 

D.   Oversight  and  Evaluation  in  the  Senate  Committee  on  Agriculture 
and  Forestry 

James  W.  Giltmier,  a  professional  staff  member  of  the  Senate 
Committee  on  Agriculture  and  Forestry,  presented  the  concluding  case 
study  of  the  afternoon.  He  described  the  legislative  need  for  evalua- 
tion information,  but  the  right  kind  of  evaluation  information.   Using 
his  own  experiences  as  well  as  examples  of  other  evaluation  studies, 
Mr.  Giltmier  indicated  that  much  evaluation  is  far  too  complicated  for 
legislative  decisionmaking  purposes.   Evaluation  methods  which  use 
systems  analysis  models  and  thoroughly  reliable  and  valid  data  may  be 
required  by  Federal  agencies  or  academics,  but  they  do  not  always  serve 
legislative  decisionmakers  who  want  to  know  whether  or  not  a  program  works. 

The  General  Accounting  Office  often  provides  important  evaluation 
and  oversight  information.   However,  Mr.  Giltmier  noted  that  some  GAO  work 
is  not  sufficiently  tailored  to  evaluation  and  oversight  purposes.   This 
occurs  either  because  the  demands  of  preparing  reports  required  by  statute 
leave  insufficient  time  for  responding  to  new  demands,  or  because  a  given 
study  inherently  requires  a  long  "turnaround  time"  and  entails  a  lengthy 
data-gathering  and  analysis  period  (as  well  as  a  seven-step  review  procedure) , 
or  because  a  study  does  not  evaluate  agency  or  program  performance  in  terms  of  the 
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"rationale  of  the  [members  of  Congress]  who  built  the  program"  but 

rather  in  terms  of  narrower  concepts  of  auditing  or  performance  efficiency. 

Mr.  Giltmier  concluded  his  presentation  by  describing  some  current 
Committee  evaluations.   These  include  an  evaluation  of  the  Farmers  Home 
Administration  to  determine  if  the  agency  has  sufficient  manpower  in 
Washington  and  in  the  field  to  implement  its  responsibilities.   Another 
evaluation  effort  consists  of  monitoring  recently  enacted  legislation 
on  forestry  which  included  congressionally  mandated  requirements  for 
evaluation.   The  General  Accounting  Office  assisted  the  committee  with 
both  evaluations,  helping  the  committee  particularly  in  drafting  legisla- 
tive requirements  for  evaluation  for  the  forestry  bill. 
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'FOREIGN  AFFAIRS  OVERSIGHT:   ROLE  OF  THE  STAFF  SURVEY  MISSION," 
JOHN  H.  SULLIVAN,  HOUSE  COMMITTEE  ON  INTERNATIONAL  RELATIONS 


It  is  a  pleasure  to  have  been  asked  to  participate  in  this  seminar 
on  Legislative  Oversight  and  Program  Evaluation.   The  subject  is  one  which 
turns  me  on.   Next  to  legislating  —  which  is  the  life-blood  of  any 
committee  —  oversight  is  the  reason  we  staff  people  get  paid. 

The  Committee  and  Oversight 

The  committee  I  come  from  —  once  the  Committee  on  Foreign  Affairs 
and  now  the  Committee  on  International  Relations  —  has  a  long  record 
of  concern  for  oversight  and  review. 

One  of  the  most  written-about  Congressional  study  missions  in  re- 
cent history  was  the  Javits  Mission  to  Europe  early  in  the  1950 's  which 
found  abuses  in  the  Marshall  Plan  programs  and  recommended  a  series  of 
reforms  most  of  which  ultimately  were  adopted. 

Some  26  years  ago  the  Committee  formed  a  special  Subcommittee  for  the 
Review  of  Foreign  Aid  Programs,  chaired  by  the  Chairman  of  the  full  Commit- 
tee with  membership  by  top  ranking  members  of  both  parties.   Most  of  the 
staff  oversight  studies  and  many  of  the  Congressional  study  missions  were 
undertaken  under  the  auspices  of  this  Review  Subcommittee.   Its  work 
resulted  in  U.S.  foreign  operations,  especially  in  the  area  of  foreign 
aid. 

More  recently,  in  this  94th  Congress,  the  Committee  revised  its  sub- 
committee structure  in  keeping  with  the  Committee  Reform  Act  of  1974  and 
new  jurisdiction  which  was  given  to  it.   Right  now  it  has  three  subcommittees 
with  essentially  an  oversight  function.   They  are: 
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The  Subcommittee  on  Oversight,  chaired  by  Chairman  Morgan.   It  deals 
with  oversight  of  foreign  intelligence;  administration  of  foreign  aid; 
external  research  —  the  General  Accounting  Office,  Congressional  Research 
Service;  Inspector  General  of  Foreign  Assistance;  Auditor  General  of  AID; 
and  all  areas  not  covered  by  other  subcommittees. 

The  Subcommittee  on  Investigations,   chaired  by  Congressman  Lee  Hamilton 
of  Indiana,  which  is  to  deal  with  special  foreign  policy-related  investiga- 
tions referred  by  the  chairman,  with  the  consent  of  the  committee.   Most 
recently  that  subcommittee  has  been  studying  the  entire  question  of  United 
States  policies  in  Indochina.   It  has  played  a  constructive  role  in  keeping 
the  committee  abreast  of  the  fast-moving  events  in  Indochina. 

The  Subcommittee  on  Future  Foreign  Policy  Research  and  Development.  This 
subcommittee  was  formed  pursuant  to  the  mandate  in  the  Committee  Reform  Act 
of  1974  that  committees  undertake  "futures  research  and  forecasting".   To  my 
mind  that  encompasses  a  large  measure  of  oversight.   One  cannot  go  about 
prescribing  for  the  future  until  one  understands  the  mistakes  and  problems 
of  the  past. 

Yet  another  manifestation  of  the  seriousness  with  which  the  Chairman 
and  members  of  the  Committee  on  International  Relations^approach  the  over- 
sight function  is  the  biannual  oversight  report  required  by  the  Legislative 
Reorganization  Act  of  1970. 

Some  committees  seemingly  have  undertaken  those  reports  as  if  they  were 
a  nuisance.  Others  have  included  oversight  information  in  their  "surveys  of 
activities"  —  thus  lumping  together  their  legislative  and  oversight  activities. 

The  Committee  on  International  Relations  —  like  the  Ways  and  Means 
Committee  and  the  Education  and  Labor  Committee  —  have  seen  the  biannual 
report  requirement  as  an  opportunity  to  pull  together  in  one  place  a 
comprehensive  account  of  oversight  activities, 
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It  has  been  my  hope  that  the  House  Committee  on  Government  Operations 
would  begin  looking  at  those  reports  and,  in  effect,  "giving  grades". 

Oversight  of  Overseas  Operations 

My  purpose  here  today  is  not  to  talk  about  the  Committee,  but  about  means 

of  conducting  Congressional  oversight  on  U.S.  operations  overseas. 

Basically,  there  are  three  ways  such  oversight  has  been  carried  out: 

First,  through  Congressional  study  missions  in  which  members  of  Congress  — 
a  delegation,  several,  or  sometimes  just  one  —  go  abroad  to  look  one  or  several 
foreign  affairs  problems. 

Second,  staff  survey  missions  in  which  professional  staff  members  of 
relevant  committees  are  sent  abroad  by  their  Committees  to  obtain  information 
on  problems  and  to  report  back  to  the  members. 

Third,  the  General   Accounting  Office  often  carries  out  studies 
which  involve  their  personnel  abroad  —  or  personnel  from  the  Washington 
office  who  go  abroad  —  in  order  to  obtain  information  for  reports  requested 
by  Congressional  committees. 

Occasionally  Congressional  Research  Service  staffers  may  become  involved 
in  such  studies.   Several  years  ago  two  staff  members  here  went  to  Europe  to 
gather  information  on  the  continuing  usefulness  of  Radio  Free  Europe  and 
Radio  Liberty  —  a  study  which  attracted  considerable  attention  when  it  was 
released. 

My  purpose  today  is  not  to  talk  about  Congressional  study  missions 
in  which  the  Members,  themselves  participate.   Not  because  I  doubt  their 
ability  to  uncover  important  information  affecting  policy.   You  might  recall 
that  it  was  just  such  a  Congressional  delegation  visiting  Vietnam  in  1960  which 
uncovered  the  infamous  "tiger  cages"  at  Con  Son  Island.   I  exclude  such 
missions  because  their  character  so  often  if  diffuse. 
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Some  are  serious  fact-finding  efforts  and  some  are  not.   Just  as  the 
American  press,  unfortunately,  is  unable  to  tell  the  difference,  discussion 
of  them  here  today  might  lead  far  afield  from  our  focus  on  oversight  activi- 
ties. 

Nor  do  I  wish  to  speak  about  GAO  and  other  activities  overseas.   They 
provide  a  valuable  back-up  to  us  in  Congress  but  they  do  not  have  —  and 
possibly  will  never  have  —  the  "inside"  impact  which  a  staff  report  can 
have. 

Before  launching  into  some  case  studies  and  examples  of  such  staff 
studies  —  both  from  the  Committee  on  International  Relations  and  the 
Senate  Committee  on  Foreign  Relations  —  let  me  make  several  preliminary 
points: 

First,  the  information  which  can  be  obtained  through  a  staff  survey 
mission  often  can  be  obtained  in  no  other  way. 

Second,  the  professional  qualifications  and  attitudes  of  those  staff 
members  undertaking  the  study  are  essential  to  their  success. 

The  staff  members  must  have  the  trust  —  or  at  least  the  respect  — 
of  the  officials  of  the  executive  branch  and  others  with  whom  they  deal.   More 
important,  they  must  have  the  trust  and  respect  of  the  members  of  Congress. 

Having  this  trust,  they  then  should  be  free  to  exercise  their  own  inde- 
pendent judgments  of  situations  —  whether  or  not  their  findings  accord  with 
the  prevailing  wisdom  of  their  committee  or  the  executive  branch. 

Third,  information  uncovered  by  a  Congressional  staff  person  and  con- 
veyed to  influential  members  of  Congress  can  be  a  powerful  force  for  changing 
the  executive  branch  policies  which  are  at  conflict  with  the  mandates  of 
Congress. 
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Moose  and  Lowenstein,  Moose  and  Meissner 

By  far  the  most  influential  staff  survey  team  has  been  Moose  and 
Lowenstein,  Dick  Moose  of  the  Senate  Foreign  Relations  Staff  and  Jim 
Lowenstein,  now  back  in  the  Department  of  State.   Dick  Moose  now  has 
a  new  partner  —  Chuck  Meissner,  formerly  of  Sen.  Percy's  staff,  now 
on  the  Foreign  Relations  Committee  staff. 

This  new  duo  seems  equally  able  to  afflict  the  bureaucratic  estab- 
lishment. 

It  was  Moose  and  Lowenstein  who  in  1973  discovered  that  the  United 
States  bombing  in  Cambodia  was  being  coordinated  from  the  United  States 
Embassy  in  Phnom  Penh.   That  revelation  burst  like  a  B-52  blockbuster  in 
Congress.   It  looked  to  many  that  Cambodia  was  a  new  tunnel  at  the  end 
of  the  light,  with  growing  U.S.  involvement.   Thus,  two  months  later 
the  Senate  passed  the  Church-Case  amendment  to  halt  the  bombing  in  that 
country. 

It  was  to  Moose  and  Lowenstein  in  the  fall  of  1973  that  President 
I  Marcos  revealed  for  the  first  time  that  an  assassination  plot  against 
1  him  has  been  uncovered  —  a  plot  which  included  the  hiring  of  American 
gunmen  to  do  the  killing.   This  information  has  probably  resulted  in 
somewhat  more  sympathy  f°r  Marco's  martial  law  decrees  on  Capitol  Hill 
than  otherwise  might  have  been  the  case. 

Most  recently,  it  was  Moose  and  Meissner  who  found  out  that  the 
United  States  Ambassador  in  Saigon,  Graham  Martin,  was  dragging  his 
feet  on  the  evacuation  of  Americans  from  Vietnam  when  it  seemed  in- 
evitable that  the  country  would  fall  to  the  Communist  advance.   They 
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found  that  planes  leaving  the  country  for  safe  havens  were  going  with 
many  seats  empty  —  though  many  persons  were  clamoring  to  leave. 

This  information  caused  senators  to  put  pressure  on  the  President  to 
give  first  priority  to  getting  the  Americans  out,  which  he  ulitmately  did. 
American  numbers  had  been  reduced  to  around  one  thousand,  four  hundred,  by 
the  time  the  final  helicopter  lift  occurred. 

But  it  is  much  too  tempting  to  emphasize  the  dramatic  aspects  of 
these  Senate  staff  missions.  Beyond  these  headline  making  aspects  of 
their  investigations  is  a  great  deal  of  solid  detail  —  much  of  it  at 
variance  with  the  official  wisdom.  As  the  Washington  Monthly  recently  said, 
the  cables  and  reports  sent  back  by  these  Senate  investigators  —  and 
I  quote  —  "provided  the  Senators  with  a  steady  stream  of  fresh,  undiluted 
information." 

House  Staff  Survey  Missions 

Although  the  staff  survey  missions  undertaken  by  the  House  Foreign 
Affairs  Committee  have  not  received  the  same  public  attention  as  those 
of  their  Senate  counter-parts,  they  had  their  moments. 

For  example,  in  the  fall  of  1972,  Jack  Brady  and  Bob  Boyer  of  our 
staff  spent  three  weeks  investigating  the  narcotics  trade  in  Southeast  Asia. 
By  small  plane  they  flew  into  the  no-man's-land  known  as  the  "Golden 
Triangle"  where  Burma,  Laos  and  Thailand  come  together. 

The  area  is  notorious  for  drug  smuggling. 

While  there  Brady  and  Boyer  uncovered  what  has  come  to  be  known  as  the 
"extra  ton  of  opium". 

The  U.S.  Bureau  of  Narcotics  and  Dangerous  Drugs  had  agreed  with 
the  Thai  Government  to  what  amounted  to  a  pre-emptive  buy  of  opium  from  the 
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Chinese  Irregular  Forces  who  have  been  occupying  northern  Thailand  for 
years  and  funding  their  activities  with  opium-growing. 

The  U.S.  turned  over  $1  million  dollars.   When  the  Chinese  Irregulars 
showed  up  at  the  turn-in  point  they  had  27  tons  of  opium  rather  than 
the  26  which  had  been  agreed  upon. 

Thai  and  U.S.  authorities  refused  to  accept  the  extra  ton  and  the 
Chinese  were  ordered  instead  to  get  it  out  of  Thailand. 

Our  men  were  highly  critical  of  that  action  —  the  expectation 
being  that  the  opium  probably  found  its  way  to  U.S.  troops  who  then  were 
still  in  Vietnam. 


On  a  1972  mission  looking  into  U.S. -funded  family  planning  programs 
in  Asia,  my  partner  —  Chips  Chester  —  and  I  uncovered  the  fact  that  sitting 
in  a  Stockton,  California  warehouse  were  625  vasectomy/lUD  surgical  kits 
valued  at  $625  thousands. 

They  once  had  been  destined  for  Korea.   But  there  was  no  way  for  the 
Koreans  to  use  them.   So  the  Agency  for  International  Development  was 
storing  them,  not  knowing  what  to  do  with  them. 

Time  does  not  permit  an  examination  of  that  particular  bureaucratic 
blunder.   Suffice  to  say  that  it  resulted  from  the  actions  of  an  official 
with  too  much  foreign  aid  money  to  spend,  and  too  little  wisdom  on  how  it 
should  be  expended. 


A  much  more  important  finding  from  that  mission  was  that  the  birth 
control  pills  which  AID  was  distributing  to  women  in  Asia  were  having 
adverse  physical  reactions  and  the  program  was  suffering  thereby. 
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The  problem  arose  because  the  kind  of  pill  distributed  each  year  depended 
on  which  American  drug  company  has  the  lowest  bill.   Each  year  for  five  years 
a  different  company  was  awarded  the  contract. 

As  a  result  a  woman  in  the  Phillipines  or  Indonesia  who  had  been  in  the 
program  for  5  years  was  switched  to  a  new  pill  every  year.   In  1972,  the  switch- 
over produced  a  very  worrisome  side  effect  —  and  women  were  dropping  out  of 
the  birth  control  program  in  droves. 

Dr.  Ravenholt  who  runs  AID's  Population  Bureau  believes  the  pill  has  no 
adverse  side  effects  —  in  fact,  is  good  for  the  health. 

He  wasn't  too  interested  in  hearing  about  these  problems  and  his  population 
officers  in  the  various  countries  were  not  too  keen  on  reporting  the  bad  news  to 
the  boss. 

But  we  heard  about  it  from  the  local  doctors  and  some  of  the  women  who  were 
taking  the  pill. 

We  described  the  problem  in  our  report  —  at  which  point  Dr.  Ravenholt  sent 
three  doctors  to  Asia  to  check  out  the  situation.   Their  report  confirmed  our 
findings. 

As  a  result,  AID  changed  its  pill  distribution  practices  to  minimize  future 
such  problems. 

Incidentially,  after  reading  that  report  one  cdlumnist  was  moved  to  call 
Chips  and  me  the  "Masters  and  Johnson  of  Capitol  Hill." 

Head  Count  in  Cambodia 

The  case  study  which  I  want  to  discuss  in  some  detail  has  to  do  with  the 
effort  by  the  congress  to  minimize  United  States  involvement  in  Cambodia. 

In  1972,  the  members  of  the  Senate  Foreign  Relations  Committee  became 
alarmed  at  what  appeared  to  be  growing  United  States  involvement  in  Cambodia. 
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As  a  means  of  dealing  with  the  issue,  the  Committee  appended  an  amendment 
to  the  Foreign  Assistance  Act  of  1972  setting  a  ceiling  of  200  on  the  number 
of  American  officials  who  could  be  in  the  country,  and  a  ceiling  of  75  third 
country  nationals  working  on  contract  with  our  mission,  aiding  the  Cambodian 
armed  forces. 

Since  the  matter  would  be  a  point  at  conference,  the  Committee  in  the 
fall  of  that  year  sent  Bob  Boyer  and  me  to  Cambodia  to  assess  the  entire 
situation  and  to  recommend  whether  the  ceiling  numbers  should  be  accepted 
and  at  what  level. 

In  the  course  of  our  investigation  we  talked  to  dozens  of  U.S.  and 
Cambodian  officials  about  the  numbers.   We  even  took  our  own  head  count  to 
find  out  how  many  official  Americans  were  in  country  at  that  time.   On  our 
own  we  walked  into  the  Air  American  office  to  count.   We  visited  a  semi- 
secret  communications  facility.   And  we  added  numbers  to  numbers. 

When  we  were  finished  we  cabled  our  members  that  a  200  ceiling  seemed 
like  a  reasonable  figure.   While  we  were  still  abroad  the  House  conferees 
yielded  to  the  Senate  on  that  point. 

In  the  spring  of  1974,  Jack  Brady  and  I  visited  Cambodia  as  part  of 
a  staff  study  of  military  and  economic  aid  in  Indochina. 

In  the  meantime  the  situation  in  the  country  had  deteriorated  and  the 
U.S.  mission  had  become  considerably  more  activist  in  its  relations  with  the 
Cambodians. 

The  200-person  ceiling  had  begun  to  pinch. 

As  a  result,  despite  the  plain  wording  or  tne  law  and  its  obvious  intent, 
we  found  that  an  average  of  215  to  222  American  civilian  and  military  officials 
were  daily  at  work  in  the  U.S.  Mission  in  Phnom  Penh  or  elsewhere  in  the  country. 

The  rationale  provided  by  the  Embassy  was  that  not  more  than  200  American 
officials  ever  were  present  in  Cambodia  overnight.   They  admitted  that  more 
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than  that  number  were  present  in  Cambodia  at  any  one  time  during  working 
hours. 

In  order  to  insure  that  no  more  than  200  official  Americans  were  present 
overnight,  the  Embassy  had  initiated  a  procedure  they  called  "headspace1.1. 

Under  "headspace'  the  Deputy  Chief  of  Mission  saw  to  it  that  sufficient 
personnel  were  out  of  the  country  by  nightfall  to  accomodate  those  on  TDY  who 
might  be  staying  overnight. 

In  many  cases  individuals  deemed  expendable  traveled  to  Bangkok  for 
several  days  of  sunning  by  the  pool,  all  the  time  being  carried  on  the  work 
rolls  and  collecting  per  diem  for  the  vacation. 

This  procedure  was  a  clear  attempt  to  circumvent  the  law  —  and  an 
expensive  one  at  that.   The  government  was  forced  to  pick  up  air  fares  both 
for  those  going  our  and  those  coming  in,  plus  salary  for- days  not  worked, 
plus  per  diem. 

When  we  confronted  the  Ambassador  with  this  scheme,  he  denied  that  it 
violated  the  law.   We  let  him  know  that  our  opinion  was  otherwise. 

While  we  were  in  the  process  of  writing  our  report  after  returning  to 
Washington,  we  received  information  that  the  Embassy  was  reducing  its  daily 
work  force  back  to  200. 

That  information  was  contained  in  our  report  to  the  Committee.   But  be- 
cause it  was  not  corroborated,  we  suggested  that  Congressional  action  might 
still  be  needed  to  remedy  the  clear  abuse  of  the  law. 

Although  no  legislative  action  was  taken,  several  congressmen  filed  a 
legal  suit  against  the  Administration  and  various  federal  officials,  citing  our 
report,  and  charging  them  with  violations  of  law. 

I  should  note  that  our  initial  information  about  the  abuse  came  from  the 
GAO  regional  director,  Fred  Lyons,  with  whom  we  regularly  meet  when  we  are  in 
that  area.   But  Fred  had  only  some  of  the  facts:   he  was  blocked  from  getting  the 
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rest  because  of  limited  access  to  the  people  and  documents  involved.   We  had 
the  access  he  lacked  and  could  complete  the  picture. 

Knowing  how  bureaucratic  processes  tend  to  take  on  a  life  of  their  own, 
the  "headspace  practice"  might  well  have  expanded  even  beyond  the  15  or  so 
extra  people  daily  —  especially  as  the  Cambodian  situation  deteriorated 
further. 

That  would  have  been  just  that  many  more  to  have  had  to  evacuate  when  the 
final  fall  of  Phnom  Penh  occurred. 

Not  long  before  that  fall  —  in  February  of  this  year  —  Jack  Brady  and  I 
made  another  visit  to  Cambodia,  as  staff  members  assigned  to  the  Congressional 
delegation  which  went  to  Indochina  at  the  behest  of  the  President. 

The  Ambassador  was  not  happy  to  see  us.  He  has  been  named  a  defendant  in 
the  legal  case.  But  he  confirmed  that  the  "headspace"  practice  had  come  to  an 
end  soon  after  our  last  visit. 

This  incident  points  up  some  of  the  points  I  made  earlier. 

Often  the  information  can  be  gotten  no  other  way  except  through  a  fact- 
finding mission  abroad.   In  this  instance,  though  someone  in  Washington  obviously 
knew  about  the  headspace  violation,  the  information  was  not  of  the  type  to  be 
volunteered  to  Congress.   It  had  to  be  found  out  in  Cambodia. 

Second,  the  experience  and  background  of  the  investigators  is  important. 
I  had  been  in  Cambodia  before  and  knew  the  law  on  the  200-man  ceiling.   Jack 
Brady  has  a  U.S.  Army  background  which  gave  him  an  understanding  that  the 
personnel  "bookeeping"  system  involved  in  "headspace"  was  an  adaptation  of  a 
military  procedure. 

Third,  the  uncovering  of  information  of  this  sort  can  result  in  changes 
of  Executive  Branch  policies  which  are  at  variance  with  Congressional  intentions. 
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Two  Additional  Points 


Before  closing,  two  additional  points  deserve  discussion. 

First,  staff  study  trips  are  best  taken  in  pairs,  or  with  three  persons. 
Solo  performances  should  be  left  to  the  members. 

Why?  Because  two  heads  being  better  than  one  is  not  just  a  musty  adage  — 
it  is  true.   Second,  for  corroboration  of  evidence.   Executive  Branch  officials 
sometimes  have  a  tendency  later  to  deny  what  they  have  told  a  Congressional 
investigator.   Rather  than  have  one  man's  word  against  another,  it  is  far 
preferable  to  have  one  man's  word  against  two  others. 

On  the  Committee  on  International  Relations  we  must,  by  Committee  practice, 
travel  in  pairs. 

In  the  Senate,  however,  some  members  of  the  Foreign  Relations  Committee 
sometimes  found  it  hard  to  understand  why  both  Moose  and  Lowensteln  had  to 
go,  when  a  single  air  fare  might  have  sufficed. 

Second,  as  a  result  of  the  Moose  and  Lowenstein  successes,  some  politi- 
cal scientists  have  suggested  that  the  Congress  establish  watchdog  staffs  in 
various  parts  of  the  world  to  provide  the  members  with  information. 

That  just  won't  work.   Staff  members  who  are  far  removed  from  the 
Committee  and  the  Congress  they  are  to  serve  will  lose  the  daily  close  con- 
tact with  members  which  is  essential  to  building  trust  and  rapport.   Such 
staffs  would  be  able  to  do  little  more  than  the  GAO  now  is  able  to  do. 

At  the  same  time,  it  might  be  advisable  for  the  relevant  committees  to 
increase  the  number  of  teams  available  for  investigations  abroad.   The 
Committee  on  International  Relations  has  acted  recently  to  strengthen  its 
capabilities  by  hiring  a  young  auditor  and  attorney  with  experience  in  re- 
viewing foreign  assistance  programs  abroad. 
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Conclusion 

In  conclusion,  It  must  be  remembered  that  the  Congressional  staff  survey 
abroad  is  only  one  of  a  number  of  ways  in  which  the  Congress  can  exercise  its 
oversight  responsibilities  in  the  area  of  foreign  affairs. 

This  oversight  process  must  be  a  continuous  one,  involving  hearings, 
studies  and  other  forms  of  research  into  executive  branch  activities  abroad. 

But  the  staff  study  has  proved  its  worth  and  is  on  its  way  to  becoming  an 
institutionalized  method  of  providing  necessary  oversight  information  to  the 
Congress. 
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DISCUSSION 

Several  points  especially  pertinent  to  the  relationship  between 
evaluation  and  oversight  were  raised  in  the  discussion  following  Dr. 
Sullivan's  paper. 

As  noted  by  a  previous  speaker,  there  is  a  difference  between 
process  and  impact  evaluation.   Process  evaluations  answer  questions 
about  the  efficiency  of  program  management  and  use  data  normally  col- 
lected in  administering  a  program.   Impact  evaluations  are  designed 
to  answer  questions  about  whether  a  program  caused  a  change  in  the 
participants'  behavior  and  or  attitudes.   Data  usually  collected  in 
the  normal  course  of  administering  a  program  are  insufficient  to  an- 
swer impact  questions.   The  differnce  between  these  two  types  of  eval- 
uations was  demonstrated  in  the  committee's  investigation  of  the  use 
of  birth  control  pills  in  Southeast  Asia.   The  Agency  for  Internation- 
al Development  collected  information  on  the  distribution  of  the  pills 
and  correlated  this  with  birthrate  data.   However  this  type  of  pro- 
cess evaluation  was  insufficient  since  it  did  not  provide  informa- 
tion to  describe  the  side-effects  of  birth  control  pills  on  the  women 
participating  in  the  program.   Thus  in  this  case,  the  process  evalu- 
ation criterion  of  efficiency,  measured  in  terms  of  distribution  of 
birth  control  pills  was  not  sufficient  to  identify  problems  relating 
to  the  adverse  impacts  of  the  program. 

It  was  also  noted  that  since  agencies  have  vested  interests  in 
maintaining  programs  and  budget,  frequently  they  are  reluctant  to 
report  problems  which  might  indicate  program  failure  or  deficiency. 
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Several  questions  were  raised  about  the  pros  and  cons  of  using 
GAO  versus  committee  staff  investigators  in  overseas  oversight  ac- 
tivities.  It  was  noted  that  GAO  investigations  sometimes  are  in- 
adequate.  Several  reasons  were  given.   First,  agency  field  per- 
sonnel may  accord  more  "weight"  or  status  to  committee  investiga- 
tive staff  than  to  GAO  staff  investigators.   Also  GAO  investigations 
usually  take  longer  to  do.   However,  frequently  GAO  investigative 
staff  develop  important  leads,  which  committee  staff  then  follow. 

Several  additional  remarks  were  made  about  the  use  of  commit- 
tee investigative  staff  for  overseas  field  work.   Dr.  Sullivan  re- 
marked that  usually  it  is  advantageous  to  detail  two  or  three  staff 
to  an  assignment,  since  it  may  be  necessary  to  have  someone  cor- 
roborate information  which  may  later  be  denied.   It  may  also  be 
useful  to  use  both  majority  and  minority  staff  in  such  investiga- 
tions since,  unlike  GAO  staff,  committee  staff  members  have  rap- 
port with  committee  chairmen  and  members  and  immediate  access  to 
them. 
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"THE  FREEDOM  OF  INFORMATION  ACT,"  WILLIAM  G.  PHILLIPS,  FORMER  STAFF 
DIRECTOR  (1971-1975) ,  FOREIGN  OPERATIONS  AND  GOVERNMENT  INFORMATION 
SUBCOMMITTEE,  HOUSE  COMMITTEE  ON  GOVERNMENT  OPERATIONS  22_/ 


I.   Introduction 


This  case  study  of  legislative  oversight  by  the  House  Foreign 
Operations  and  Government  Information  Subcommittee  40/  of  the  Govern- 
ment Operations  Committee  represents  a  unique  situation  in  several 
respects.   First,  it  involves  a  subcommittee's  activities  where  the 
same  Government  Operations  Committee  unit  has  both  oversight  and 
legislative  jurisdiction,  and  is  one  of  the  few  cases  where  the  Gov- 
ernment Operations  Committee  exercises  such  broad  substantive  leg- 
islative authority.   Moreover,  the  case  study,  which  spans  a  period 
of  3  1/2  years,  began  with  an  intensive  series  of  oversight  inves- 
tigations and  hearings  on  the  administration  of  the  Freedom  of  Infor- 
mation Act  (5  U.S.C.  552),  and  proceeded  through  every  step  of  the 
legislative  process  as  well  —  hearings  on  recommended  amendments, 
committee  action,  floor  action  and  passage  in  the  House  and  Senate, 
and  finally  culminating  in  their  enactment  into  law  over  a  Presi- 
dential veto. 


39/   For  a  detailed  legislative  history  of  the  1974  amendments  to  the 
Freedom  of  Information  Act,  see  Joint  Committee  Print,  House  Gov- 
ernment Operations  Committee  and  Senate  Judiciary  Committee,  "Free- 
dom of  Information  Act  and  Amendments  of  1974  (P.L.  93-502)." 
Washington:   U.S.  Government  Printing  Office,  March,  1975. 

40/  The  name  of  the  subcommittee  was  changed  to  "Government  Informa- 
tion and  Individual  Rights"  at  the  beginning  of  the  94th  Congress 
(January  1975) . 
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The  Freedom  of  Information  Act  of  1966  (FOIA)  (P.L.  89-487)  ^jj 
was  a  milestone  enactment,  providing  that  "any  person"  might  have  ac- 
cess (with  certain  exceptions)  to  rules,  opinions,  orders,  and  other 
types  of  public  records  of  various  agencies  of  the  executive  branch  of 
the  Federal  Government.   The  product  of  more  than  a  decade  of  investiga- 
tions, studies,  reports,  and  vigorous  advocacy  for  the  "people's  right 
to  know,"  the  1966  law  was  an  amendment  to  section  3  of  the  Administra- 
tive Procedure  Act  of  1946.   It  reversed  long-standing  information  pol- 
icies and  practices  of  the  Federal  executive  bureaucracy  by  establish- 
ing the  right  of  public  access  (including  corporate  entities)  to  various 
types  of  identifiable  Federal  agency  records  not  falling  within  the 
nine  categories  of  exemptions  specified  in  the  act.   It  gave  the  public 
the  right  to  sue  in  Federal  court  to  obtain  the  records.   There  was 
no  requirement  that  a  person  be  required  to  state  his  reasons  for  want- 
ing the  information  and  it  further  required  that  the  burden  of  proving 
the  legal  justification  for  withholding  of  requested  records  in  each 
case  be  the  responsibility  of  the  Federal  agency  involved. 

Withholding  of  information  by  a  Federal  agency  must  be  justified 
on  the  basis  of  one  or  more  of  the  nine  specific  exemptions  permitted 
by  section  552(b)  of  the  act.   They  relate  to  matters  that  are: 

(1)  Specifically  required  by  Executive  order  to  be  kept  secret 
in  the  interest  of  the  national  defense  or  foreign  policy; 

(2)  Related  solely  to  the  internal  personnel  rules  and  practices 
of  an  agency; 


41/  Codified  as  Section  552,  title  5,  U.S.  Code  by  P.L.  90-23,  with 
slight  modifications,  on  June  5,  1967. 
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(3)  Specifically  exempted  from  disclosure  by  statute; 

(4)  Trade  secrets  and  commercial  or  financial  information  ob- 
tained from  a  person  and  privileged  or  confidential; 

(5)  Inter-agency  or  intra-agency  memorandums  or  letters  which 
would  not  be  available  by  law  to  a  party  other  than  an  agency 
in  litigation  with  the  agency; 

(6)  Personnel  and  medical  files  and  similar  files  the  dis- 
closure of  which  would  constitute  a  clearly  unwarranted  in- 
vasion of  personal  privacy; 

(7)  Investigatory  files  compiled  for  law  enforcement  purposes 
except  to  the  extent  available  by  law  to  a  party  other  than  an 
agency; 

(8)  Contained  in  or  related  to  examination,  operating,  or  con- 
dition report  prepared  by,  on  behalf  of,  or  for  the  use  of  an 
agency  responsible  for  the  regulation  or  supervision  of  finan- 
cial institutions;  or 

(9)  Geological  and  geophysical  information  and  data,  including 
maps,  concerning  wells. 

The  act  makes  it  clear  in  section  552(c)  that  the  exemptions  have 

absolutely  no  effect  upon  congressional  access  to  information: 

This  section  does  not  authorize  withholding  of  information  or 
limit  the  availability  or  records  to  the  public,  except  as  spe- 
cifically stated  in  this  section.   This  section  is  not  authority 
to  withhold  information  from  Congress. 

Investigative  efforts  on  information  policies  and  practices  of 
Federal  agencies  were  initiated  in  1955  by  the  subcommittee's  predeces- 
sor—  the  Special  Subcommittee  on  Government  Information.   They  were 
spearheaded  in  the  House  by  Representative  John  E.  Moss  (D.,  Calif.), 
its  chairman  until  1971.   Similar  activities  were  carried  on  in  the 
Senate  by  the  Administrative  Practice  and  Procedure  Subcommittee  of 
the  Judiciary  Committee,  first  under  the  chairmanship  of  Senator  Thomas 
Hennings  (D.,  Mo.),  and  later  by  Senator  Edward  V.  Long  (D.,  Mo.). 

Voluminous  House  and  Senate  hearings,  staff  studies,  and  commit- 
tee reports  disclosed  the  widespread  "secrecy  practices"  employed  by 
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the  executive  bureaucracy.   Increasing  attention  was  also  being  di- 
rected at  abuses  of  Government  secrecy  by  the  news  media,  its  pro- 
fessional organizations,  and  distinguished  students  of  journalism 
such  as  the  late  Harold  Cross.   Under  the  guise  of  "national  security1 
or  the  cloak  of  "administrative  anonymity,"  vast  quantities  of  infor- 
mation generated  by  Federal  agencies  were  hidden  from  public  scru- 
tiny, often  to  avoid  political  embarrassment,  bureaucratic  mistakes, 
administrative  decisions  contrary  to  the  public  interest,  cost  over- 
runs, or  to  cover  up  conflicts  of  interest,  or  corruption. 

When  congressional  hearings  were  held  in  the  Johnson  adminis- 
tration on  legislation  to  correct  these  abuses,   executive  branch 
witnesses  were  unanimous  in  their  opposition.   Freedom  of  Informa- 
tion legislation,  they  contended,  would  destroy  the  effective  op- 
eration of  Federal  agencies  by  bogging  them  down  in  processing  vast 
numbers  of  requests  for  information  from  citizens  and  the  news  me- 
dia.  Even  after  Congress  had  unanimously  passed  the  bi-partisan 
bill  which  subsequently  became  the  Freedom  of  Information  Act  in 
1966,  secrecy-minded  bureaucrats  pleaded  with  President  Lyndon 
Johnson  to  veto  the  measure.   But  Johnson  ignored  these  pleadings 
and  signed  the  Freedom  of  Information  bill  into  law  on  July  4, 
1966,  stating  that  "a  democracy  works  best  when  the  people  have 
all  the  information  that  the  security  of  the  Nation  permits."  It 
became  effective  one  year  later —  July  4,  1967. 

Another  unique  aspect  of  this  milestone  law  is  that  each  Fed- 
eral agency  —  presently  more  than  100  —  administers  the  provisions 
of  the  FOIA  within  the  agency,  based  on  its  own  agency  regulations 
promulgated  in  the  Federal  Register  and  based  on  legal  interpretative 
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guidelines  of  the  Attorney  General.   Only  minimum  consultative  as- 
sistance is  provided  to  agencies  by  the  Justice  Department's  Office 
of  Legal  Counsel  through  its  Freedom  of  Information  Committee,  es- 
tablished as  an  informal  information  clearinghouse  in  December,  1969. 
This  means  that  congressional  oversight  over  the  administration  of 
the  FOIA  must  necessarily  involve  scrutiny  of  the  information  ac- 
tivities of  each  of  the  more  than  100  Federal  agencies  subject  to 
its  provisions  —  a  most  tedious,  time-consuming  task  for  the  hand- 
ful of  House  and  Senate  staff  personnel  assigned  the  task  by  the 
two  subcommittees  having  jurisdiction  over  the  act. 

II.   House  Oversight  Plan  Formulated  in  1971 

During  the  four  years  following  the  effective  date  of  the  FOIA, 
House  and  Senate  subcommittees  had  engaged  in  only  sporadic,  limited 
types  of  oversight  activity.   Initial  agency  regulations  to  implement 
the  act  were  reviewed  for  compliance  with  congressional  intent  and  a 
compilation  was  published  as  a  House  Committee  Print .   The  Senate  com- 
mittee had  compiled  a  10  month  review  of  the  operation  of  the  new  law. 
Both  staffs  had  on  numerous  occasions  acted  as  "informal  ombudsmen" 
in  providing  information  and  assistance  to  congressional  offices,  his- 
torians, the  news  media,  and  the  general  public  involving  requests  to 
agencies  for  Government  information  under  the  FOIA. 

The  reorganization  of  the  House  Government  Operations  Committee 
at  the  beginning  of  the  92nd  Congress  resulted  in  the  appointment  of 
a  new  subcommittee  chairman,  Representative  William  S.  Moorhead  (D., 
Pa.).   One  of  the  priority  targets  agreed  upon  by  the  subcommittee  was 
a  thorough  investigation  of  the  operation  of  the  four  year  old  FOIA, 
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which  had  then  operated  for  approximately  two  years  under  a  Democratic 
(Johnson)  administration  and  two  years  under  a  Republican  (Nixon)  ad- 
ministration. 

The  FOIA  oversight  plan,  as  formulated  in  the  spring  of  1971,  in- 
volved several  stages  to  take  full  advantage  of  dramatic  events  that 
focused  on  the  latest  public  disclosures  of  widespread  Government  se- 
crecy.  Late  in  June,  the  subcommittee  commenced  public  hearings  on  the 
various  issues  of  Government  information  policies  and  withholding  prac- 
tices involved  in  the  so-called  "Pentagon  Papers"  controversy.   These 
hearings  and  follow-up  subcommittee  inquiries  in  1972  focused  on  clas- 
sification operations  within  the  executive  branch  under  the  security 
classification  system  established  by  President  Eisenhower  in  1953  (Ex- 
ecutive Order  10501) .   These  hearings  also  explored  the  relationship 
of  such  "classified"  types  of  information  to  the  FOIA  exemption,  legal 
and  constitutional  questions  involved  in  publication  of  the  "Pentagon 
Papers"  by  the  New  York  Times  and  Washington  Post,  and  matters  of  pub- 
lic policy  in  the  field  of  Government  information. 

The  next  stage  was  the  extensive  preparation  for  in-depth  over- 
sight hearings  by  the  subcommittee  on  the  administration  of  the  FOIA 
by  Federal  agencies.   A  detailed  questionnaire  was  formulated  by  mid- 
summer 1971  and  was  sent  to  each  of  the  Federal  agencies  subject  to 
the  FOIA.   The  questionnaire  sought  voluminous  types  of  statistical 
records  from  the  agencies  — 

*  How  many  formal  requests  were  made  for  information  under  the 
act? 

*  How  many  requests  were  granted?  How  many  denied?  Which  of 
the  nine  exemptions  was  involved  in  each  denial?  How  many 
were  granted  after  intra-agency  appeal  processes?  How  many 
such  cases  ended  up  in  the  courts?  What  was  the  outcome  in 
each  case? 
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*  What  types  of  groups,  corporations,  or  individuals  made  re- 
quests for  information  under  the  FOIA?  How  many  requests 
for  each  category  of  requester? 

*  How  does  the  security  classification  system  of  the  agency  func- 
tion under  Executive  Order  10501?  How  many  officials  are  au- 
thorized to  classify  information  under  each  of  the  three  clas- 
sification levels? 

The  data  requested  was  to  cover  the  period  of  the  effective  date  of 
the  FOIA  (July  4,  1967)  through  July  4,  1971.   Before  the  question- 
naire was  formally  sent  to  executive  branch  agencies,  it  was  "pre- 
tested" by  discussing  possible  questions  with  a  number  of  Government 
officials  who  would  have  the  eventual  responsibility  of  responding 
to  the  questionnaire.   Because  of  the  volume  of  data  requested,  a  3- 
month  period  was  provided  for  agency  replies .   All  but  a  few  agen- 
cies complied,  although  several  claimed  they  had  kept  no  such  rec- 
ords on  the  FOIA. 

The  expert  assistance  of  the  Congressional  Research  Service, 
Library  of  Congress,  was  solicited  at  an  early  stage  in  the  for- 
mulation of  the  oversight  plan.  Responses  to  the  questionnaire  were 
analyzed  and  tabulated  by  Dr.  Harold  Relyea  and  Ms.  Sharon  Gressle 
of  the  Government  and  General  Research  Division  of  CRS.   Other  re- 
lated research  papers,  case  analyses,  and  background  data  were  also 
prepared.   Additional  expert  consultant  assistance  in  this  project 
was  also  provided  by  CRS  to  the  subcommittee  staff  and  Members. 

The  final-  analyses  and  tabulations  of  data  provided  by  agencies 
in  response  to  the  subcommittee  questionnaire  were  completed  by  CRS 
personnel  early  in  1972  and  preparations  for  oversight  hearings  in 
the  spring  began.   A  group  of  representative  executive  agencies  were 
selected  to  appear  at  the  hearings  on  the  basis  of  the  questionnaire 
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analysis.   They  included  old-line  departments,  newer  departments,  in- 
dependent regulatory  agencies,  boards,  military  services,  executive 
agencies,  commissions  —  a  broad  cross-section  of  the  executive  bu- 
reaucracy. 

Those  with  better  than  average  records  of  compliance  with  con- 
gressional intent  in  the  administration  of  the  FOIA  were  included, 
along  with  other  agencies  whose  records  of  administering  the  FOIA 
were  a  negative  quantity.   Subcommittee  staff  personnel  spent  sev- 
eral weeks  in  individual  conferences  with  the  key  officials  of  de- 
partments and  agencies  to  be  called  to  testify  at  the  oversight 
hearings.   Additional  information  to  supplement  questionnaire  data 
was  obtained  in  this  way.   Information  provided  by  the  agencies 
was  also  clarified  in  some  cases;  dates  of  appearance  and  names  of 
the  most  knowledgeable  agency  officials  to  testify  were  likewise 
negotiated  during  this  informal  staff  consultative  process.   Exten- 
sive efforts  were  also  made  by  the  subcommittee  staff  to  solicit 
the  testimony  of  certain  key  non-governmental  witnesses  who  could 
provide  expert  testimony  from  their  vantage  point  on  the  positive 
and  negative  aspects  of  agencies'  administration  of  the  FOIA. 

The  subcommittee's  public  oversight  hearings  on  the  administra- 
tion of  the  act  consumed  fourteen  days  of  intensive  inquiry  during 
the  months  of  March  and  April,  1972.   Some  fifty  witnesses  testified, 
including  spokesmen  for  a  broad  cross-section  of  executive  depart- 
ments and  agencies,  plus  news  media  representatives,  attorneys  ex- 
perienced in  FOIA  litigation,  former  Government  officials  involved 
in  FOIA  matters,  academicians,  and  representatives  of  organizations 
utilizing  the  FOIA. 
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The  next  stage  of  the  subcommittee's  oversight  work  took  place 
during  the  months  of  May  and  June,  19  72.   Additional  hearings  were 
held  on  special  aspects  of  the  FOIA's  operations,  including  several 
days  on  the  functioning  of  the  security  classification  system  in  the 
executive  branch  under  Executive  Order  10501  and  the  ways  in  which 
that  system  would  be  affected  by  the  new  Executive  Order  11652,  is- 
sued by  President  Nixon  in  March,  1972.   Other  hearings  were  held  on 
the  issues  involving  the  right  of  Congress  to  information  from  the 
executive  branch  and  on  the  information  functions  of  the  approximately 
2,000  Federal  advisory  committees  which  exert  tremendous  influence  on 
governmental  policy  decisions  —  mostly  in  meetings  held  behind  closed 
doors. 

All  told,  the  1971-1972  oversight  hearings  of  the  subcommittee 
—  spanning  a  12-month  period  —  involved  41  days  of  public  hearing; 
testimony  was  received  from  142  witnesses  who  furnished  9  volumes  of 
valuable  information  on  every  conceivable  aspect  of  operations  of  the 
FOIA  and  matters  relating  to  the  dissemination,  as  well  as  the  with- 
holding, of  information  by  executive  agencies  of  the  Federal  Govern- 
ment. 

With  the  conclusion  of  the  subcommittee's  series  of  FOIA  over- 
sight hearings  in  June,  1972,  work  began  immediately  on  the  next 
stage  of  the  plan  —  the  drafting  and  consideration  of  a  comprehen- 
sive investigative  report  based  on  these  hearings.   The  initial  staff 
draft  was  completed  during  the  summer,  edited  and  revised  during  bi- 
partisan staff  meetings,  and  circulated  among  members  of  the  subcom- 
mittee.  It  was  further  revised  during  subcommittee  mark-up  sessions, 
unanimously  approved,  and  forwarded  to  the  full  Government  Operations 
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Committee  membership  for  further  consideration.   There  it  was  discussed, 
further  amended,  and  unanimously  adopted  by  the  full  committee  late  in 
September,  1972.   The  90-page  report  was  subsequently  issued  as  H.  Rept. 
92-1419.   ("Administration  of  the  Freedom  of  Information  Act"). 

The  report  described  the  major  problem  areas  in  the  operation  of 
the  act,  documenting  each  by  specific  examples  from  the  hearing  record. 
They  included:    (1)  bureaucratic  delays  in  responding  to  requests  for 
information;  (2)  abuses  of  fee  schedules  for  search  and  copying  of  re- 
quested records;  (3)  cumbersome  and  costly  legal  remedies  to  contest 
agency  withholding  of  requested  information;  (4)  the  lack  of  involve- 
ment of  public  information  personnel  within  agencies  in  the  decision- 
making process  affecting  disclosure  or  withholding  of  requested  records; 
(5)  the  relative  lack  of  use  of  FOIA  by  the  news  media;  and  (6)  the 
lack  of  a  more  positive  attitude  toward  FOIA  by  top-level  administra- 
tors of  executive  agencies  and  departments. 

The  report  also  made  a  series  of  recommendations  to  the  agencies 
to  improve  their  administration  of  the  act  and  set  forth  a  separate 
series  of  recommendations  calling  for  substantive  amendments  to  the 
FOIA  requiring  further  legislative  consideration  by  the  committee.  Bi- 
partisan bills  were  introduced  by  Chairman  Moorhead  and  other  members 
to  implement  these  legislative  recommendations  during  the  next  several 
months.   This  led  into  the  next  stage  of  the  subcommittee's  plan  — 
legislative  hearings  on  proposed  amendments  to  strengthen  and  improve 
the  operation  of  the  FOIA. 

Five  days  of  legislative  hearings  were  held  on  such  bills  by  the 
subcommittee  during  the  first  two  weeks  in  May,  1973.   Executive  branch 
witnesses  flatly  opposed  any  amendments  to  strengthen  the  FOIA.   However, 
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the  legislation  was  supported  by  the  American  Bar  Association's  Ad- 
ministrative Law  Section,  news  media  witnesses,  and  many  other  or- 
ganizations and  individuals  familiar  with  bureaucratic  subversion  of 
congressional  intent  of  the  original  FOIA. 

Subcommittee  mark-up  sessions  continued  through  the  fall  in  an 
effort  to  obtain  a  bi-partisan  consensus  on  a  package  of  workable 
amendments  to  the  act  but  final  agreement  was  not  reached  until  early 
the  following  year.  A  clean  bill  representing  the  bi-partisan  com- 
promises reached  during  the  subcommittee  mark-up  meetings  was  introduced 
in  January,  1974  and  the  measure  was  cleared  for  action  by  the  full 
Government  Operations  Committee. 

The  bill  (H.R.  12471)  was  considered  by  the  full  committee  and 
unanimously  reported  to  the  House  on  February  21,  1974.   (H.  Rept.  93- 
876).   It  contained  more  than  a  dozen  substantive  amendments  to  the 
act  to  correct  the  major  deficiencies  in  its  operation  described  in 
the  committee's  investigative  report.   A  few  weeks  later  the  bill  was 
debated  on  the  House  floor  and  passed,  with  minor  technical  amendments, 
by  a  vote  of  383-8  and  sent  to  the  Senate. 

Meanwhile,  during  the  spring  of  19  73,  the  Administrative  Practice 
and  Procedure  Subcommittee  of  the  Senate  Judiciary  Committee,  chaired  by 
Senator  Edward  M.  Kennedy  (D.,  Mass.),  and  two  Senate  Government  Oper- 
ations Committee  subcommittees  having  overlapping  jurisdiction  in  the 
field  of  the  FOIA  and  Government  secrecy  generally  had  also  held  exten- 
sive joint  hearings  in  this  same  area.   Some  11  days  of  hearings  were 
held  and  testimony  was  received  from  over  40  witnesses,  amounting  to  3 
volumes  of  printed  proceedings.   The  Senate  version  of  the  1974  FOIA 
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amendments,  quite  similar  to  the  House  bill,  was  favorably  reported 
early  in  May  and  passed  the  Senate,  with  amendments  by  a  vote  of  64- 
17  on  May  30th. 

The  House-Senate  conference  committee  met  in  public  sessions  on 
4  days  during  the  month  of  August,  1974  to  resolve  differences  between 
the  two  bills.   The  meetings  overlapped  the  resignation  of  President 
Nixon  and  the  swearing-in  of  Vice  President  Ford  as  President.   The  new 
President  requested  that  the  conferees  defer  further  conference  action 
until  he  had  an  opportunity  to  study  the  legislation  and  to  submit  his 
views.   The  request  was  honored  and  President  Ford  subsequently  trans- 
mitted a  letter  to  the  conference  committee  detailing  several  major  ob- 
jections to  specific  provisions  of  the  measure.   The  conferees  later 
agreed  to  several  important  compromises  in  an  effort  to  reach  an  ac- 
commodation of  conflicting  executive  and  legislative  views.   Conference 
agreement  was  finally  reached  in  September,  1974  and  the  conference  re- 
port was  routinely  adopted  early  in  October  by  both  the  House  and  Senate 
and  the  bill  was  sent  to  the  White  House. 

Just  prior  to  the  congressional  recess  for  the  November  elections, 
President  Ford  surprisingly  vetoed  the  legislation.   Virtually  every 
executive  agency  had  recommended  such  action,  just  as  the  bureaucracy 
had  recommended  to  President  Johnson  that  he  veto  the  original  1966 
law.   But  the  Ford  veto,  on  the  heels  of  his  public  pledge  to  advance 
the  cause  of  "open  Government"  was  unexpected.   The  dramatic  unfolding 
of  the  sordid  details  of  the  Watergate  coverup,  paralleling  congres- 
sional consideration  of  the  FOIA  amendments  in  1973  and  1974,  had  ob- 
viously helped  to  persuade  Congress  and  the  public  of  the  need  for  a 
stronger  law  to  protect  the  public's  "right  to  know". 
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The  Presidential  veto  of  the  19  74  FOIA  amendments  was  considered 
by  the  House  on  November  20th  and  overriden  by  a  vote  of  371-31.  The 
Senate  followed  suit  the  next  day  by  a  65-27  vote  and  the  bill  became 
P.L.  93-502.  The  amendments  became  effective  on  February  19,  1975  — 
90  days   after  enactment.      42/ 

III.      Some   General  Observations 

General  observations  below  on  the   role  of  a  congressional  subcom- 
mittee or  committee  in  its   exercise  of  positive  oversight   and/or  pro- 
gram evaluation  are  based  on  both   the  case  history   of   the   1974  amend- 
ments  to   the  Freedom  of   Information  Act  as  set   forth  above  and   on 
extensive  work  of  the  Foreign  Operations    and  Government   Information 
Subcommittee  in  the   foreign  operations   area  of   its   jurisdiction  during 
the  same  1971-1975  period. 

Much  of  the   subcommittee's  activity  in   this   part   of   its   jurisdic- 
tion was   concentrated  on  the  operation  of  various   types   of  U.S.    assis- 
tance to   South  Vietnam,    Laos,    and   Cambodia  in   the   late   1960's   and  early 
1970' s   through  programs   administered  by  the  Agency  for   International 
Development    (AID) . 

It  held  extensive  investigative  hearings   into  such   subject   areas 
as   the  operation  of   the   illegal  black-market   in   these   countries,   nar- 
cotics  trading,    the   smuggling  of   gold,    diversion  of  U.S.    funds   and 
property,    the  so-called  "Phoenix"  program,    refugee   relief,   health   pro- 
grams,   the  U.S.    propaganda  assistance  provided  the  Government  of  South 


42/     Details   of   the   17  major  substantive   amendments   to  the   1966   act  in- 
corporated in  P.L.   93-502   are   discussed  at  pages   113-116   of   the 
Joint  Committee  Print    (March,    1975)    cited  above. 
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Vietnam  (GVN)  by  the  USIA,  the  conduct  of  the  South  Vietnamese  na- 
tional elections,  public  works  assistance,  currency  stabilization 
programs,  and  many  other  non-military  assistance  efforts  provided 
by  the  U.S.  to  the  then  existing  Governments  of  these  three  coun- 
tries. 

During  these  investigations,  the  subcommittee  utilized  a  num- 
ber of  outside  investigative  sources.   They  included  field  work  by 
General  Accounting  Office  personnel  in  Vietnam  on  the  "pacification" 
program  that  resulted  in  a  report  and  subsequent  testimony  at  sub- 
committee hearings  on  the  subject  by  top  officials  of  the  GAO's 
International  Division.   Other  information  was  obtained  through 
cooperative  agreements  at  the  staff  level  with  personnel  of  the 
House  Foreign  Affairs  and  Senate  Foreign  Relations  Committee.   On 
a  number  of  occasions,  valuable  leads  for  investigative  follow-up 
were  obtained  from  "whistle-blowers"  with  first-hand  information 
about  what  they  felt  were  violations  of  congressional  intent  or 
outright  corruption  in  the  administration  of  various  AID  programs 
in  these  countries.   Other  useful  data  was  obtained  by  subcommittee 
staff  investigators  through  interrogations  of  agency  personnel  or 
through  information  furnished  by  AID  witnesses  called  to  testify 
on  specific  allegations  or  on  general  program  activity  being  con- 
ducted in  those  countries. 

Throughout  the  foreign  operations  portion  of  the  subcommittee's 
oversight  functions  during  this  period,  there  was  an  "adversary"  re- 
lationship with  the  executive  agency.   Members  and  staff  became  wary 
of  the  value  of  official  information  furnished  by  AID  officials  be- 
cause of  the  many  examples  of  inaccurate  or  misleading  data  provided 
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during  hearings  or  on  request  to  the  agency.   During  one  of  the  sub- 
committee's investigation  of  the  AID  program  in  Cambodia,  President 
Nixon  exercised  "Executive  privilege"  to  withhold  documents  that  had 
been  requested  from  the  agency,  documents  of  the  type  that  had  been  rou- 
tinely furnished  the  subcommittee  on  a  regular  basis  involving  many 
other  countries  (including  Cambodia)  for  many  years. 

Perhaps  one  logical  explanation  of  this  "adversary"  position  is 
that  less  executive-legislative  branch  cooperation  can  usually  be  ex- 
pected when  the  Federal  Government  is  operating  during  a  period  of 
divided  responsibility  —  one  branch  controlled  by  one  political  party 
and  one  by  the  opposite  party  —  as  has  been  the  situation  during  the 
past  7  years.  Many  of  these  same  observations  can  be  made  with  re- 
spect to  the  adversary  relationships  between  the  subcommittee  and 
executive  agencies  during  the  oversight  and  legislative  activities 
involving  the  Freedom  of  Information  Act. 

One  effective  technique  utilized  by  the  subcommittee  during  its 
investigation  of  AID  program  activity  in  Laos  involved  the  use  of  a 
member  of  the  staff  as  the  lead-off  witness  to  put  into  precise  con- 
text the  scope  and  detail  of  staff  investive  efforts.  By  narrative 
summary,  he  described  in  testimony  the  ways  in  which  the  various  pro- 
grams functioned,  presented  evidence  of  waste  and  maladministration 
he  had  uncovered,  and  set  the  stage  for  subsequent  cross-examination 
of  responsible  AID  officials  at  another  subcommittee  hearing. 

Congressional  oversight  might  well  be  divided  into  three  main 
categories  — 

1.   "Routine"  oversight,  for  example,  would  be  the  day-to-day 
monitoring  of  agency  regulations  as  published  in  the  Federal  Register 
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to  implement  the  1974  FOIA  amendments  —  contacting  responsible  agency 
personnel  to  call  attention  to  technical  errors,  misinterpretations  of 
congressional  intent,  or  to  request  clarifying  amendments  to  the  regu- 
lations. 

Another  example  of  a  "routine"  type  of  oversight  would  be  informal 
briefing  sessions  of  subcommittee  members  and/or  staff  with  officials 
of  an  agency  to  become  more  familiar  with  a  specific  program,  to  be- 
come better  acquainted  with  agency  personnel,  to  clarify  a  policy  or 
operational  change  of  the  agency,  or  as  a  preliminary  step  prior  to 
the  scheduling  of  oversight  hearings  with  the  particular  agency.   Un- 
less there  was  a  major  controversy  involved,  the  subsequent  hearing 
with  agency  officials  would  also  be  considered  as  "routine"  oversight. 

2.   "Special"  oversight  differs  from  "routine"  in  that  it  usually 
represents  a  major  effort  of  the  subcommittee  in  an  important  area  of 
its  jurisdiction  that  has  a  high  priority.   Special  oversight  work 
also  indicates  the  need  for  a  long-term  series  of  investigations,  hear- 
ings, sometimes  requiring  the  use  of  expert  consultants,  General  Ac- 
counting Office  investigative  work,  staff  liaison  with  the  staffs  of 
other  House  and  Senate  committees  having  collateral  jurisdiction,  field 
investigative  trips,  and  other  types  of  in-depth  studies.   It  also 
means  that  the  results  of  such  "special"  oversight  will  not  only  be 
valuable  as  part  of  a  major  investigative  report  of  the  committee, 
but  are  also  likely  to  include  longer-range  legislative  recommendations 
that  relate  to  amendments  to  statutes  falling  within  the  committee's 
jurisdiction.   Subsequent  legislative  actions  to  implement  these  recom- 
mendations would  also  be  a  part  of  the  overall  plan. 
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The  case  study  of  the  19  74  Freedom  of  Information  Act  amendments 
covering  more  than  3  years  of  oversight  and  legislative  activity  by 
the  subcommittee  would  be  a  prime  example  of  such  "special"  oversight. 

3.   "Guerilla"  oversight,  the  final  category  to  be  discussed 
here,  is  —  as  the  title  implies  —  a  type  of  "spot  checking"  of  mis- 
cellaneous activities  by  agencies  coming  within  the  scope  of  the  sub- 
committee's jurisdiction.   It  differs  from  the  type  of  "routine"  con- 
tacts with  agency  officials  mentioned  above  in  that  it  will  more  likely 
represent  an  adversary  or  conflict  situation.   Often,  the  impromptu 
visit  of  a  staff  investigator  to  the  office  of  an  agency  official,  or 
a  telephone  call  to  discuss  some  agency  program,  decision,  or  other 
action  of  interest  to  the  subcommittee  can  have  a  very  salutary  ef- 
fect. 

If  additional  impact  is  considered  necessary,  the  subcommittee 
chairman  might  call  a  "1-day"  hearing  to  follow-up  the  preliminary 
contact  and  to  dramatize  the  agency  shortcoming  in  that  particular 
case.  The  resulting  publicity  may  prompt  quick  remedial  action  by 
the  agency.  A  formal  committee  report  may  then  be  issued  to  focus 
added  attention  of  the  Congress  and  the  public  on  the  agency's  in- 
ept behavior. 
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DISCUSSION 

Several  important  points  were  raised  in  the  discussion  following  Mr. 
Phillips'  presentation.   In  summary  these  are: 

(1)  Good  oversight,  and  the  information  gathering  and  analytical 
skills  which  underlay  good  oversight,  require  very  competent  professional 
staff; 

(2)  Good  oversight  generally  requires  the  adoption  of  an  adversary 
stance  between  the  executive  and  legislative  branches  regardless  of  the 
bipartisanship  of  the  issue  or  the  fact  that  the  President  and  the 
majority  in  Congress  may  be  of  the  same  party; 

(3)  An  analytical  research  organization  which  ascribes  to  the 
ethics  of  objectivity  and  non-partisanship  generally  cannot  perform  all 
the  tasks  required  for  good  oversight,  since  oversight  has  to  take  place 
in  the  political  arena.   The  appropriate  role  for  CRS  is  as  an  information 
resource  not  an  evaluator.   Generally  the  smaller  the  congressional 
committee  oversight  staff,  the  larger  the  CRS  assistance  role  in  oversight. 

(4)  The  credibility  of  an  investigative  staff  person  is  usually 
not  diminshed  if  he  or  she  delivers  testimony  before  the  congressional 
committee.   However,  investigative  staffers  should  be  able  to  weigh 
the  information  gathered  according  to  both  majority  and  minority  points 
of  view  if  these  views  differ. 

(5)  Adequate  routine  oversight  can  be  obtained  frequently  through 
normal  responses  to  constituent  inquiries. 

(6)  Oversight  would  be  improved  if  committee  staff  gave  more  atten- 
tion to  the  Federal  regulations  that  agencies  issue.   It  is  important 

to  assess  whether  the  regulations  correspond  to  the  intent  of  the  Congress 
in  passing  the  mandating  legislation  for  a  program.   Usually  not  enough 

attention  is  paid  to  regulations  printed  in  the  Federal  Register. 
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"CONGRESSIONAL  COMMITTEE  CASE  STUDIES:   STUDIES  IN  PUBLIC  WELFARE" 
ALAIR  A.  TOWNSEND,  FORMER  RESEARCH  DIRECTOR  OF  THE  SUBCOMMITTE 
ON  FISCAL  POLICY,  JOINT  ECONOMIC  COMMITTEE. 


Thank  you  for  the  opportunity  to  discuss  with  you  how  and  why  the 
Subcommittee  on  Fiscal  Policy  conducted  a  comprehensive  public  welfare  st'  i; 
that  resulted  in  the  24-volume  series  entitled  Studies  in  Public  Welfare. 
Origin  of  the  Study 

The  focus  and  nature  of  our  study  was  shaped  by  a  legislative  debate, 
out  6f  which  it  grew  directly.   The  debate  was  over  the  Family  Assistance 
Plan,  the  Presidential  proposal  first  advanced  in  1969  to  establish  a  Federal 
minimum  income  floor  for  all  families  with  children.   As  you  know,  former 
Representative  Martha  Griffiths,  who  was  chairman  of  the  subcommittee,  also 
was  a  senior  member  of  the  House  Ways  and  Means  Committee,  which  handles  a 
substantial  portion  of  public  welfare  legislation. 53/  The  deliberations  o~ 
the  Ways  and  Means  Committee  in  1970  and  1971  on  the  Family  Assistance  P__.., 
known  was  FAP,  convinced  Mrs.  Griffiths  and  others  of  the  necessity  to  a. ...  „yi;e 
FAP  and  related  plans  in  the  context  of  all  the  existing  income-Lasted  c.i...) 
assistance  programs,  the  in-kind  programs  offering  food,  health,  day  care 
and  housing  benefits,  and  the  social  insurance  programs- —  such  ;:3  social 


*  Ms.  Townsend  was  Research  Director  of  the  Subconmittee  en  Fiscal  Policy 
during  its  study  of  public  welfare  programs.   She  is  not.-  Senior  Analyst  - 
Human  Resources  for  the  House  Budget  Committee.   Views  express./!  are  the 
author's  only,  and  do  not  necessarily  reflect  those  of  the  Housj  Budget  '.  . 
mittee,  its  members  or  staff. 

53/For  Mrs.  Griffiths'  own  statement  of  her  concerns  in  conducting  the  st,  ., 
see  attachment  1,  her  "Foreword"  to  income  Security  for  A-icric.ins.   Her 
statement  of  appreciation  also  shows  the  range  of  assistance  received  cxi\. 
resources  utilized  in  the  course  of  tlie  study. 
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security  and  unemployment  insurance  —  rather  than  simply  in  the  context 
of  the  existing  public  assistance  programs.  In  the  FAP  debate  attention 
was  directed  to  the  following  problem  areas. 

1.  Adequacy  of  Benefits.   First,  in  order  to  assess  the  adequacy  of  FAP 
benefit  levels,  policy-makers  had  to  know  what  other  benefits  and  private 
income  were  available  to  potential  FAP  recipients  and  how  they  were  dis- 
tributed.  Lacking  this  information,  one  could  not  know  the  combined  benefit 
and  income  levels  that  were  implicit  in  the  legislation.   For  example,  the 
original  FAP  basic  cash  benefit  level,  $1,600  annually  per  family  of  four 

in  1969,  appeared  to  be  cruelly  low  to  some,  but  for  many  recipients  this 
was  to  be  supplemented  by  State  welfare  payments  and  by  the  benefits  thac 
usually  are  attached  to  enrollment  in  the  program  of  Aid  to  Families  with 
Dependent  Children  (AFDC).   These  fringe  benefits  include  free  medical  care, 
free  school  lunches  for  the  children,  eligibility  for  food  stamps  or  surplus 
food  commodities,  and,  often,  free  day  care.   Depending  on  the  State  in 
which  it  lives,  a  family  of  four  with  no  other  income  now  can  receive  a 
package  of  benefits  that  would  require  the  equivalent  of  more  than  $7,500  in 
gross  earnings  (that  is,  earnings  before  taxes  and  work  expenses  are  paid) 
to  purchase.   Of  course,  AFDC  families  in  many  areas  receive  nowhere  near 
this  amount,  but  during  the  FAP  debate  no  one  was  very  certain  who  received 
what  because  benefits  in  the  form  of  food,  health,  and  housing  rarely  are 
included  in  official  census-style  surveys,  and  measurements  of  the  extent 
of  poverty  do  not  count  the  value  of  such  benefits  in  summing  the  incomes 
of  the  poor. 

2.  Work  Incentives.   Another  reason  for  evaluating  refor.n  plans  such  as 

FAP  in  a  broad  programmatic  context  is  that  the  level  of  combined  cash,  fooo, 
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housing,  health,  and  other  benefits  is  only  one  factor  in  determining  the 
effect  of  such  plans  on  work  incentive.   Equally  important  is  the  way  in 
which  benefits  are  tailored  to  income. 

Today,  many  programs  have  sliding  scales  of  benefits  that  fall  as 
income  rises.   Some  of  these  benefit-reduction  or  benefit-loss  rates  are 
high  and  some  are  low.   Unemployment  insurance  typically  has  a  100  percent 
benefit-loss  rate — that  is,  a  dollar  earned  is  a  dollar  of  unemployment 
compensation  lost.   AFDC  benefits  generally  are  reduced  by  67  cents  for  each 
dollar  of  net  earnings.   As  income  rises  by  $1.00,  you  must  pay  about  30 
cents  more  for  the  same  amount  of  food  stamps  and  about  25  cents  more  in 
public  housing  rent.   If  income  rises  above  a  certain  level,  you  may  lose 
eligibility  for  medicaid  completely.   The  cost  of  day  care  may  increase  if 
the  center  bases  its  fee  schedule  on  family  income.   The  basic  opportunity 
grant  program  also  provides  scholarships  based  on  family  income,  and  the 
higher  the  family's  income,  the  lower  the  aid. 

The  number  of  programs  based  on  need  seems  to  be  increasing,  and,  thus, 

the  number  of  low-  or  even  middle- income  families  eligible  for  multi-benefits. 

Since  each  program  takes  its  share  of  increased  income,  the  cumulative 

effect  sometimes  is  that  recipients  lose  more  than  a  dollar  in  benefits 

when  they  earn  an  extra  dollar. 

If  simply  superimposed  on  existing  programs,  new  proposals  worsen  this 

problem.   The  Family  Assistance  Plan  suffered  fatal  attack  after  it  became 
clear  that  it  both  raised  the  guarantee  level  for  penniless  persons  receiving 
various  combinations  of  benefits,  and  more  importantly,  raised  to  extraordinarily 
high  levels  the  rate  at  which  combined  benefits  would  have  to  be  surrendered 
if  earnings  rose.   The  work  incentives  so  carefully  built  into  t"AP  would  neve 
been  undermined  by  the  program's  interaction  with  others. 
3.   Equity  of  Benefits.   The  equity  of  benefits  became  another  issue  in  the 
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FAP  debate.   The  distribution  of  existing  benefits,  especially  those  tied 
to  eligibility  for  AFDC,  was  unfair.   Most  of  the  cash  aid  was  unavailable 
to  "employables,"  no  matter  how  poor  they  were.   Fully  employed  fathers, 
for  instance,  were  ineligible  for  AFDC,  even  though  such  families  were  a 
significant  prart  of  the  poverty  population.   The  Family  Assistance  Plan, 
by  opening  up  Federal  cash  aid  to  intact  families,  promised  to  reduce  the 
discrimination  against  male-headed  families.   But,  critics  countered,  as  long 
as  FAP  benefits  were  lower  than  the  AFDC  benefits  provided  by  some  States, 
and  as  long  as  eligibility  for  other  benefits  such  as  medicaid  was  tied  to 
AFDC,  inequities  could  be  reduced  but  not  eliminated. 

That  criticism  was  valid  but  it  was  not  generally  appreciated  how 
much  male-headed  families  already  were  receiving  in  benefits  to  supplement 
low  income.   In  fact,  some  States  and  localities  were  supplementing  the 
earnings  of  working  fathers  under  their  own  general  assistance  or  home 
relief  programs;  such  working  men  were  eligible  for  the  food  stamp  program, 
which  acquired  national  eligibility  rules  during  the  FAP  debate  and  expanded 
to  cover  most  of  the  Nation.   Intact  families  were  eligible  for  public 
housing  and  other  subsidized  housing  programs  on  a  space  available  basis 
like  everyone  else;  and  in  some  States  they — or  their  children — were 
eligible  for  free  medical  care  under  medicaid.   If  they,  were  employable  but 
unemployed — whether  fully  or  partially — many  were  eligible  for  unemployment 
insurance  or  for  the  unemployed  father  portion  of  AFDC,  available  in  about 
half  the  States.   However,  the  extent  to  which  such  programs  helped  to  reduce 
the  inequities  created  by  AFDC  or  potentially  residual  under  FAP  was 
unknown  because  little  data  existed  on  who  was  benefiting  from  these  programs. 
Thus,  to  judge  the  effect  of  FAP  on  equity  of  benefits  as  well  as  their 
adequacy,  new  data  were  needed. 
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A.  Administration  and  Program  Planning.   The  problem  of  administering 
this  complex  set  of  programs  came  into  focus  in  the  FAP  debate.   Rapid 
expansion  of  such  programs  as  AFDC  and  food  stamps  was  outpacing  the  ability 
of  administrators  to  handle  them  within  existing  organizational  structures. 
Administrative  confusion  was  created  by  the  complex,  ever-changing  rules 
and  eligibility  requirements.   Result:   rather  substantial  numbers  of  in- 
eligible persons  received  benefits  while  some  eligible  persons  were  rejected. 
While  each  program  had  its  own  problems,  administration  of  multiple  programs 
at  the  local  level  generally  was  uncoordinated.   A  single  family  could  be 
affected  by  several  programs,  none  of  which  shared  information  or  adminis- 
trative burdens.   The  result  was  inefficiency  and  confusion.   Rather  than 
cutting  and  paring,  it  appeared  to  many  that  FAP  was  only  adding  another 
layer.   They  wondered:   "Can't  we  develop  a  plan  that  can  consolidate  and 
substitute  for  some  of  those  that  currently  exist?" 

5.   Need  for  Overview.   Despite  these  complex  interactions  among  our  growing 
body  of  programs  designed  to  affect  standards  of  living,  neither  the  structure 
of  Congress  nor  of  the  Executive  agencies  facilitates  their  analysis.   Con- 
siderable analytical  work  is  done  by  congressional  committees,  the  Executive 
agencies,  and  nongovernmental  researchers  before  major  legislation  is  changed. 
But  in  general  such  analysis  focuses  exclusively  on  one  program  at  a  time — 
the  one  under  consideration.   Rarely  is  the  set  of  programs  reviewed  in  its 
entirety.   The  result  is  that  intended  changes  in  one  program  often  are 
countered  by  features  of  an  overlapping  program.   For  instance,  the  work- 
refusal  penalty  of  the  AFDC  program — a  reduction  in  cash  benefits — is  largely 
cancelled,  in  some  States,  by  higher  food  stamp  benefits. 

The  Congress  and  the  Executive  agencies  are  organized  in  programmatic 
terms  and  have  jurisdictional  boundaries.   For  example,  if  we  look  only  at 
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the  21  major  income  maintenance  programs  with  some  Federal  funding,  we  find 
they  fall  under  the  jurisdiction  of  10  committees  of  the  House  and  9  of  the 
Senate,  and  they  are  administered  by  11  Federal  agencies.  No  single  com- 
mittee has  the  duty  to  view  them  all,  even  though  the  programs  reach  many 
of  the  same  persons.  Each  passes  legislation  affecting  directly  only  a  few 
programs  with  different  eligibility  requirements,  different  benefit  levels, 
different  treatment  of  income. 

Proliferation  and  fragmentation  of  programs  have  created  serious 
technical  and  policy  problems,  but  they  went  unrecognized  until  the  FAP 
struggle.   Once  the  FAP  debate  raised  such  issues  in  the  Congress,  it  ap- 
peared that  one  committee  was  well  suited  to  review  the  entire  complex  of 
programs  cutting  across  many  jurisdictions.   This  was  the  Joint  Economic 
Committee,  which  is  non-legislative  and  has  a  broad  study  mandate.   Mrs. 
Griffiths  and  the  Subcommittee  on  Fiscal  Policy  received  a  separate  appro- 
priation to  conduct  this  study,  and  report  findings  to  the  full  Congress. 
The  major  rationale  for  this  study  was  to  determine  ways  to  integrate  and 
rationalize  this  growing  array  of  income-related  benefit  programs. 

To  summarize  the  background  of  the  subcommittee's  study,  our  task 
was  to  go  beyond  the  one-at-a-time  program  review  generally  performed.  We 
were  to  review  the  entire  set  of  income  support  programs  and  report  to  the 
Congress  on  the  effects  of  these  programs  with  regard  to: 

1.  The  distribution  of  benefits,  especially  in  various  combinations; 

2.  Questions  of  adequacy; 

3.  The  impact  on  incentives  with  respect  to  family  structure  and 
work  effort;  and 

A.   Administrative  efficiency. 
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The  Product 

The  study  lasted  for  three  and  one-half  years.   Its  product  was  twenty- 
four  volumes  of  staff-level  studies,  four  volumes  of  hearings,  a  summary 
report  with  recommendations  endorsed  by  the  majority  of  subcommittee  members, 
a  reform  proposal  in  bill  form,  and  numerous  articles  published  under  other 
auspices.   Several  of  the  most  innovative  projects  had  been  conceived  in 
advance  by  Chairman  Griffiths.   She  knew  well  what  the  information  gaps 
were  for  policy-makers,  and  had  outlined  in  some  detail  particular  studies 
necessary  to  fill  them.   Throughout  the  study,  Mrs.  Griffiths'  guidance  and 
practical  knowledge  of  programs,  issues,  and  how  to  present  information 
were  extremely  important  to  staff.   Her  accessibility  and  direct  participa- 
tion in  all  phases  were  key  elements  in  maintaining  policy  relevance. 

While  the  chairman  had  presented  a  working  staff  study  plan  to  sub- 
committee members  in  the  early  stage,  many  projects  simply  evolved  during 
the  study  period.   There  was  sufficient  flexibility  and  time  to  permit 
follow- through  on  good  ideas. 
Resources 

Studies  of  the  scope  intended  here,  on  complex  programs,  require 
commitment  of  resources,  time,  and  patience.   It  takes  time  to  assemble  a 
staff  and  to  collect  and  compile  information  to  do  a  thorough  jcb.   In  the 
case  of  the  subcommittee,  the  initial  funding  was  for  two  years,  extended  to 
three,  then  to  three  and  one-half  years. 

The  staff  was  a  carefully  selected  mixture  of  a  variety  of  skills  ar.d 
disciplines.   It  was  essential,  for  example,  that  there  be  several  persons 
with  strong  institutional  knowledge,  who  knew  in  detail  the  laws  and 
regulations  under  which  the  major  programs  were  administered,  and  who  also 
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knew  how  the  programs  actually  operated  in  the  field,  since  frequently  there 
is  a  wide  gulf  between  formal  rules  and  their  application.  Similarly,  we 
needed  persons  with  hard  research  and  quantitative  skills,  who  could  help 
design  surveys  and  analyze  data.  The  staff  approached  most  projects  as  a 
team,  and  the  various  skills  and  perspectives  of  individual  staff  members 
strengthened  each  project. 

A  strong  staff  was  essential  in  directing  and  reviewing  work  done  by 
outside  researchers  free  or  under  contract,  and  in  monitoring  projects 
undertaken  for  us  by  Executive  agencies,  CRS,  and  GAO.   Proper  control  of 
a  project,  in  my  view,  requires  a  strong  committee  staff — not  to  control 
findings,  but  to  retain  the  desired  focus.   These  "outside"  (non-committee 
staff)  resources  are  not  limitless,  and  should  not  be  squandered  by  lack 
of  direction. 

Other  than  staff,  our  major  resources  were  the  GAO,  CRS,  outside 
researchers,  State  and  local  program  agencies,  the  House  Information  Systems 
staff,  and  Executive  agencies.   In  any  given  project  that  involved  non-staff 
resources,  the  approach  was  to  have  a  staff  specification  of  the  project  and 
the  help  required,  followed  by  regular  monitoring. 

One  of  our  major  projects,  for  example,  involved  our  staff,  the  GAO, 
and  the  House  Information  Systems  staff  (HIS) .   The  goal  was  to  learn  more 
about  the  distribution  of  benefits,  and  combined  benefits  in  particular. 
To  do  this,  the  staff  and  GAO  chose  six  low-income  study  sites.   Utilizing 
city  block  maps  and  other  available  resources,  the  GAO  selected,  in  each 
site,  a  random  sample  of  about  300  households.   Combining  use  of  telephone 
and  post  office  directories  with  a  good  deal  of  imagination,  -GAO  staff 
identified  the  members  of  the  sample  households.   Then,  largely  without  con- 
tacting the  households,  GAO  searched  the  records  of  over  100  Federal,  State, 
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and  local  programs  to  determine  whether  any  household  members  were  partici- 
pating.  They  combined  these  facts  by  household  to  produce  a  program 
participation  profile.   (The  results  were  dramatic.   Findings  showed  one 
family  benefiting  from  11  programs;  many  households  were  participating  in 
three  to  five  programs.)  GAO  staff  coded  and  keypunched  the  data,  House 
Information  Systems  processed  it,  and  our  staff  wrote  the  report. 54/  All 
along  the  way  there  was  interchange  between  GAO  and  subcommittee  staff,  and 
the  division  of  labor  wo-rked  well. 

In  another  case,  we  wanted  to  learn  more  about  the  actual  rules  applied 
by  States  and  localities  in  programs  such  as  Aid  .to  Families  with  Dependent 
Children,  supplementation  for  the  aged,  blind,  and  disabled,  public  housing, 
general  assistance,  and  medicaid.   These  are  programs  in  which  States  and 
localities  have  considerable  discretion.   Additionally,  we  wanted  to  see 
what  benefit  packages  might  be  available  to  various  family  types  at  various 
income  levels  in  areas  across  the  Nation.   Such  information  did  not  exist. 
So  we  devised  a  questionnaire  to  be  filled  out  by  State  agencies  (or  their 
local  designees)  on  programs  operating  in  100  counties  selected  on  a 
stratified  random  sample  basis  to  be  representative  of  the  Nation.   All 
relevant  assumptions  were  given,  such  as  age  and  sex  of  family  members,  their 
work  expenses,  their  income,  and  so  forth.   Ten  family  types  and  about  6  income 
levels  were  specified.   Before  sending  out  the  questionnaire,  our  staff  com- 
puted and  filled  in  Federal,  State,  and  local  income  taxes  and  social  security 
taxes  if  applicable;  Federal  benefits  such  as  social  security  and  veterans' 


54/  See  Paper  No.  6  as  listed  on  attachment  2. 
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pensions  if  applicable;  unemployment  insurance  benefits  (with  the  help  of 
the  Department  of  Labor);  public  housing  benefits  (with  the  help  of  the 
Department  of  Housing  and  Urban  Development);  and  so  on.   Completed 
questionnaires  were  reviewed  by  staff,  processed  by  House  Information 
Systems,  and  reports  written  by  our  staff .55/  These  two  reports — one  covering 
programs  for  the  aged  and  the  second  for  the  non-aged  —  were  widely  cir- 
culated within  Congress.   Detailed  information  on  programs  in  their 
counties  or  States  proved  to  be  of  great  interest  to  Members. 

Our  concern  with  the  problems  generated  by  insufficient  program 
coordination  naturally  led  to  thinking  about  how  to  improve  matters.   In 
conjunction  with  the  Institute  for 'Research  on  Poverty  of  the  University 
of  Wisconsin,  the  subcommittee  sponsored  a  week  long  conference  on  program 
coordination  attended  by  subcommittee  staff,  institute  researchers,  other 
experts  in  the  field,  and,  for  several  days,  Mrs.  Griffiths.   Mot;t  partici- 
pants were  asked  to  write  papers  about  specific  existing  programs,  later 
edited  and  published  by  the  subcommittee. 56/  To  assure  maximum  usefulness 
to  the  subcommittee  staff  in  developing  reform  options  for  the  chairman, 
the  papers  were  to  cover  at  least  the  following  topics:   a  description  of 
the  program  as  it  currently  operated;  the  role  it  played  in  the  income 
maintenance  system;  how  well  or  how  poorly  it  meshed  with  other  programs; 
and  optimal  restructuring  required  under  each  of  three  specific  major 
welfare  options  —  a  deraogrant,  an  employment-based  program,  and  a  negative 
income  tax-type  plan  such  as  the  Family  Assistance  Plan. 


55/  See  Paper  No.  10  and  Paper  No.  15. 

56/  See  Paper  No.  4,  Paper  No.  5  (Part  2),  Paper  No.  7,  Paper  No.  9,  and 
Paper  No.  19. 
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Another  use  of  outside  resources  involved  research  and  evaluation 
studies  of  manpower  training  programs.   Many  such  studies  of  program  effectiveness 
had  been  conducted,  largely  funded  by  the. Departments  of  Labor  and  HEW.   But 
a  variety  of  methodologies  was  involved,  findings  sometimes  were  contradictory, 
and  no  general  summary  was  available.   One  subcommittee  staff  member  under- 
took the  task  of  critically  reviewing  each  study  to  determine  the  overall 

impact  of  these  programs  on  the  poor. 57/ 

A  final  type  of  project  involving  non-staff  resources  was  compiling 

a  handbook  of  public  welfare  programs.   Very  early  in  the  study,  we 

recognized  the  need  to  have  in  one  volume, in  easily  accessible  form,  detailed 

information  on  the  rules  under  which  current  programs  operate,  especially  those 

features  that  were  important  in  considering  questions  of  program  coordination. 

So  we  developed  a  standard  format  to  be  used  for  each  of  the  major  cash  and 

in-kind  programs  that  would  specify  such  details  as  conditions  of  eligibility; 

the  eligible  units  (e.g.,  primary  beneficiary  only,  household,  family,  etc.); 

definition  of  personal  income  relevant  for  program  purposes;  hov*  that  income 

is  counted  in  determining  eligibility  and  benefit  amounts;  State 

variations  in  eligibility  and  benefit  provisions,  and  other  such  items.   Our 

primary  sources,  of  course,  were  the  law  and  regulations.   The  result  was  a 

volume  heavily  used  by  our  own  staff  and,  we  are  told,  by  many  others. 58/ 

Programs  change  constantly,  however,  and  portions  of  the  Handbook 

were  out  of  date  as  soon  as  it  was  published.   At  the  end  of  the  study  we  wantta 

to  leave  an  updated  version,  but  our  time  and  resources  were  limited. 

Staff  revised  our  fairly  detailed  format.    Staff,  CRS,  and 

the  Executive  agencies  updated  the  Handbook,  using  this  outline  as  a  mode] .59/ 

57/See  Paper  No.  3. 
58/See  Paper  No.  2. 
59/Sec  Paper  No.  20. 
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These  examples  of  our  collaborative  efforts  were  intended  to  make 
two  points.   First,  there  is  no  substitute  for  a  highly  professional  staff  that 
knows  the  substance  and  programs  under  review.    Second,  outside  resources  will 
be  best  utilized  when  the  staff  has  clearly  specified  what  it  wants  done 
and  follows  the  project  closely. 

As  I  have  mentioned,  we  made  extensive  use  of  the  published  (and  in 
some  cases,  unpublished)  data  from  Executive  agencies  and  the  considerable 
program  expertise  of  Executive  staff.    I  think  it  fair  to  say,  however,  that 
the  original  data  we  collected  on  programs  made  it  a  more  than  even 
trade.    In  several  cases,  agencies  collected  new  data  on  their  programs 
and  participants  directly  at  Mrs.  Griffiths'  request.   xhe  Department  of 
Agriculture,  for  example,  undertook  the  first  in-depth  survey  of  the 
income  and  other  benefits  received  by  households  participating  in  the  food  stamp 
and  surplus  commodity  programs. 60/ 

Presentation  of  Findings 

Doing  a  good  study  is  one  thing;  presenting  the  information  in  useful 
formats  is  another.   The. work  has  to  be  sufficiently  technical  and 
scholarly  to  be  credible  among  researchers,  academicians;  and  experts  in  the 
field.   But,  as  congressional  staff,  these  groups  were  not  our  principal 
audience.   To  reach  Members,  their  staffs,  and  committees,  it  was  necessary 
to  summarize  results  in  non-technical  terms.   Every  subject  matter  has  its 
own  technical  jargon,  which  can  be  a  barrier  to  capable,  interested  persons 
who  should  not  have  to  take  the  time  to  learn  it.   Since  it  is  too  much  to 
expect  each  long,  detailed  report  to  be  read  in  its  entirety  by  each 


60/  See  Paper  No.  17. 
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interested. person  (although  I  feel  strongly  that  the  details  have  to  be  there), 
we  made  use  of  chart  books,  detailed  press  releases  generally  sent  out  with 
publications  to  non-press  as  well  as  press,  and  summary  volumes.   We  had 
published  two  extensive  volumes  on  the  subject  of  work  incentives,  for 
example.—''   But  the  subject  was  too  important  not  to  attempt  a  short, 
summary  volume  with  many  charts  and  graphs. 62/ 

SUHHAgY 

To  summarize,  in  the  case  of  the  subcommittee,  there  were  three  elements 
for  a  successful  study:   a  chairman  interested  in  and  knowledgeable  about  the 
subject  matter,  sufficient  time  to  do  a  thorough  job,  and  a  strong  (though 
not  large)  staff. 

It  is  too  early  and  quite  difficult  to  determine  the  impact  the 
study  has  had  or  will  have  on  legislation.   Two  facts  are  relevant:    first, 
many  Members,  committees,  and  staff  of  both  houses,  are  familiar  with  the 
study;  second,  the  study  volumes  were  widely  distributed  and  even  used  as 
textbooks   in  a  range  of  disciplines.   In  terms  of  helping  to  fill  information 
gaps,  those  are  large  steps.    Moreover,  there  is  much  wider  recognition 
that  individual  cash,  food,  housing,  health,  day  care,  and  scholarship 
programs  are  simply  pieces  of  a  larger  income  maintenance'  system.     There 
is  a  new  appreciation  of  the  difficulty  of  reconciling  work  incentives  with 
adequacy  of  benefits  in  income  support  operations.     If  it  be  true  that 
recognizing  and  measuring  a  problem  is  the  first  step  on  the  road  to  solution, - 
the  subcommittee  study  has  made  a  substantial  contribution  to  a  better  way 
of  helping  the  poor. 


61/See  Paper  No.  A  and  Paper  No.  13. 
62 /See  Paper  No.  14. 
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ATTACHMENT  1 


FOREWORD 


Before  I  close  (lie  books  on  this  survey  of  income  maintenance  pro- 
grains  and  how  they  work,  I  would  like  to  take,  this  last  opportunity 
to  thank  those  who  have  so  kindly  helped  me.  cither  to  do  the  work  or 
to  secure  the  appropriations  which  made  the  study  possible.  The  Con- 
gressmen are:  Wright  Patman  of  Texas,  Wilbur  1).  Mills  of  Arkansas, 
ex-Congressman  Johnny  Byrnes  of  "Wisconsin.  George  H.  Mahon  of 
Texas,  the  late  Frank  Bow  of  Ohio,  Gerald  R.  Ford  of  Michigan;  tho 
Senators  are:  William  Proxmire  of  Wisconsin,  Jacob  K.  Javits  of 
New  York,  Abraham  A.  Ribicolf  of  Connecticut,  and  Finest  F.  Rol- 
lings of  South  Carolina.  I  want  expressly  to  thank  the  subcommit- 
tee memhers  for  (heir  encouragement  in  (he  effort  and  the  subcommit- 
tee stalf,  both  current  and  former  members,  for  the  excellence  of  their 
work,  their  patience  and  their  good  humor.  It  was  a  pleasure  to  work 
with  them. 

In  addition,  this  study  would  never  have  been  possible  without  the 
devoted  and" untiring  efforts  of  my  favorite,  agency,  the  General  Ac- 
counting Oilicc.  nor  could  this  study  have  been  completed  without  tho 
tremendous  help  of  welfare  directors  and  workers  throughout  the 
country,  particularly  those  who  aided  us  in  the  survey  of  benefits  in 
100  count  ies.  and  the  work  of  those  researchers  who  submitted  papers. 
I  want  also  to  acknowledge  with  gratitude  the  work  of  the  following 
persons  and  organizations:  Robert  llaveman,  director  of  the  Institute 
for  Research  on  Poverty,  University  of  Wisconsin:  Irene  Lurie.  co- 
ordinator of  a  Conference  on  Coordinating  Income  Maintenance  Pro- 
grams, sponsored  jointly  by  the  subcommittee  and  I  he  Institute  for 
Research  on  Poverty;  the  IIouso  Information  Systems  staff;  the  Con- 
gressional Research  Service,  particularly  Robert  Boslick  who  pre- 
pared charts  for  the  subcommittee,  and  the.  Education  and  Public 
Welfare.  Division:  Donald  Craver  of  the  Government  Printing  Oflicc, 
who  expertly  handled  the  printing  of  subcommittee  volumes;  stall's-Of 
tho  Departments  of  Health,  Education,  and  Welfare:  Agriculture: 
Housing  and  Urban  Development;  and  Labor;  Robert  Lampman  and 
Harold  Walls  of  (he  University  of  Wisconsin:  Robert  Harris.  Deo 
Bawden.  and  Gary  Hendricks  of  the  Urban  Institute;  Nelson  Mc- 
Clmig.  Treasury  Department:  David  Kershaw.  Mathematical  Theo- 
dore Marmor.  University  of  Chicago;  John  Stark,  executive  director 
of  tho  Joint  Economic  Committee:  and  James  Knowles.  former  re- 
search director  of  tho  Joint  Economic  Committee.  My  deep  apprecia- 
tion goes  to  all  of  those  named  and  many  moro  who  aided  me  in  com- 
pleting this  historic  work. 

Perhaps  the  first  question  that  should  be  answered  is  why  (his  is  (ho 
first  congressional  Study  of  welfare  and  related  programs.  That  ques- 
tion I  cannot  answer.  1  oitlv  know  why  I  believed  that  one  was  neces- 
sary and  why  1  undertook  it.  '  ■  i 

(V) 
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VI 


When  I  became  a  member  of  the  Ways  and  Means  Committee  in 

lP(i2.  I  was  first  struck  by  I  lie  incredible  inequities  of  social  security, 
which  returned  on  the  same  tax  payment  a  greater  sum  to  a  man-sup- 
ported family  than  to  a  woman-supported  family  and  more,  to  many 
families  where,  the  woman  remained  at  home  than  to  families  whero 
both  husband  ami  wife  worked,  although  the  latter  families  might  in 
.fact  have  paid  more  tax  than  one-worker  families. 

Years  ago  I  held  hearings  on  private  pensions,  and  retirees  drilled 
into  my  head  the  fact  that  through  quirks  in  the.  labor  contract  or 
oAif  right  dishonesty  in  the  pension  setup,  they  had  been  deprived  of 
their  pensions.  Of  course,  the  Nation  had  been  deprived  of  the  tax 
money  that  should  have  been  paid  on  those  pensions,  and  now  the, 
.  Nation  maintained  these  people  on  welfare. 

I  watched  far  whatever  else  I  could  find  on  the  law's  strange 
workings. 

Some,  years  airo,  I  received  a  letter  from  a  woman  in  my  district 
who  earned  Sfi.^OO  gross  per  year.  She  outlined  her  taxes,  her  take- 
home  pay  and  her  problems.  The.  next  day  T  received  a  call  from  a 
woman  living  in  a  far  better  area  of  my  district  whose  home  had  been 
purchased  under  section  235  and  who  was  drawing,  she  said,w£750  per 
month,  untaxed,  in  AFDO  money,  including  $-2<>0  per  month  for  a 
housekeeper.  My  "tired  lady",  earning  $5.-".00.  was  paying  taxes  to 
he!]-)  support  a  woman  at  a  $7.">0  rate  per  month.  T  seethed.  There  are 
no  jobs  of  which  I  am  aware  that  increase  your  pay  because  you  have 
children. 

Shortly  after  that  day  T  began  to  investigate  the  possibility  of 
making  a  complete  survey  of  all  income  maintenance  programs. 

"When  the  staff  had  been  assembled  I  was  invited  to  make  a  speech, 
on  welfare  to  a  church  group  in  Detroit.  T  asked  my  staff  to  tell  me 
what  a  woman  with  three  children  could  get,  in  welfare,  food  stamps, 
medicaid,  and  other  supplements  in  my  city.  Detroit,  and  in  New 
York,  Chicago.  Atlanta,  and  Los  Angeles;  then  to  find  the  median 
earnings  of  a  woman  in  those  cities.  We  found,  of  course,  that  the 
woman  on  welfare  did  better  in  every  city  than  the  woman  Avho 
worked  at  the  median  wage. 

I  will  never  forget  some  of  these  hostile  questions  in  that  church: 
''But  the  "woman  must  pay  some  money  for  tbe  food  stamps."  Answer: 
"We  have  considered  that.  She  is  still  getting  a  substantial  amount  of 
free  food."  Question:  "But  she  waits  all  day  for  a  doctor."  Answer: 
"Who  doesn't?  But  the.  woman  who  works  may  give  np  a  day's  pay, 
wait  all  day.  pay  :?lo  to  the  doctor  and  $50  for  prescriptions."  Ques- 
tion: "But  she  has  to  take  three  buses  to  iret  to  the  place  to  <rct  the 
food  stamps,  welfare,  etc."  Answer:  "The  woman  who  works  may 
take  three  buses  twice  a  day,  5  days  a  week  to  get  back  and  forth  to 
work." 

The.  theory  of  comparing  what  is  given  in  welfare  with  what  is 
needed  is  foolish.  "What  is  needed"  is  a  phony  standard  set  np  by  a 
paternalistic  middle,  class.  The  real  standard  is  what,  similar  people 
earn,  and  how  they  are  treated.  Few  have  ever  asked  what  those,  who 
work  need.  Those  who  have  bled  for  people  who  have  nothing  have 
not  demanded  that  people- who  work  be  treated  at  least  as  well  as 
people  on  welfare. 
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VII 


In  the  coming  effort  to  correct  the  "welfare"  injustices  there  will 
be  those  who  say  that  however  much  is  offered  is  too  little,  and  those 
who  say  that  however  little  is  offered  is  too  nnich.  T  urge  boih  groups 
to  consider  the  real  world  and  to  aslc  themselves:  "Can  we  all'ord  this 
for  all  in  like  circumstances,  or  ain  I,  today,  selecting  out  one  small 
group?" 

Personally,  I  am  for  the  law  applying  equally  to  all,  and  this  is  the 
purpose  of  the  entire  investigation  of  income  maintenance  programs. 

This  work  began  because  of  the  inequity  in  the  law.  It  is  dedicated 
to  equality  in  justice. 

Martha  "W.  Grietitiis, 
Chabinan,  Subcommittee  on  Fiscal  Policy. 
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ATTACHMENT  2 


PUBLICATIONS  OF  TUB  SUBCOMMITTEE  ON 
FISCAL  POLICY 


Papers  in  I  be  Series  :8tmHes  fothe  Public  Welfare 

PaiKT  N...    I.  "Public    Income  Tr.-in-.fer    Pre-grants;  The    incieleiiee-  of  Millli|il.! 
He-iicnls  :in«l  tin-  l--ue-  llai-.-.l  liv  Tin  ir  lle.-.-ipt."  A|ir.  III.  l'.IT'J. 

l*:ip.-r  X...  J.  "Il:iiull I;  r.r  Public  Income-  Trni.-f.-r  Program-,"  Oct,. III.    l'.IT'J. 

Paper  X...  3.  "Tli.-  i:ir.'.Miv„ f  Manp.oe.-r  Training  Program-:  A  P.e-vicw  „f 

Research  on  ili<-  ln.|.:>n  on  tin-  l'.».r."  Nov.  JO,  l'.IT'J. 
Paper  Xo.   1.  "Income-  Trai.-fe-r  Pro-rains:  How  Tlu-v  Tax  the;  Poor,"  Deo.  22, 

1*172. 
Paper  No.  ...  I-<u.-  in  Welfare-  Administration: 

(Part   1)  "Welfare— An  AdminMraiivc  Nightmare,"  Dec.  :il,  1072. 

("Part  2)  "Inti-raoviTiiiiii-inal  lSi-lailoii.hips,"  Mar.  12,  PIT:!. 

(Pnrl_3)  "Implication-  of  the  Income  Maintenance  l-Ape-rliiienls,"  Mar.   12, 

Paper   Xo.   C    "II. .n-    Public    Welfare-   Benefit*   Are    Distributed   in    I.ow-Incomu 

Areas,"  Mar.  2(i,  HIT:!. 
"Additional    Material  for  Paper  Xo.  (i:   How   Public  Welfare   Benefit*  Are   Di* 

tribuled  in  ijrtoliicomc  Ana-."  Aug.  (i,  HIT3. 
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DISCUSSION 

During  the  discussion  it  was  noted  that  the  Subcommittee  was  sub- 
ject to  some  interest  group  pressure  while  doing  the  study.   The  type  and 
intensity  of  outside  attention  varied  depending  upon  the  nature  of  the 
issue  or  program  being  studied.   Some  programs  were  very  popular  and 
study  of  them  generated  resentment,  for  instance  on  the  work  training 
programs.   Not  all  outside  group  pressure  was  necessarily  dysfunctional. 
Usually  the  committee  did  not  encounter  external  criticism.   This  may 
have  been  due  to  the  fact  that  the  committee  did  not  have  legislative 
authority.   Another  reason  may  have  been  the  fact  that  rigorous  research 
and  analytical  methodologies  were  used,  averting  possible  criticisms  about 
inadequate  quantitative  research  designs  and  analysis. 

Several  points  were  raised  about  the  administrative  costs  and  bene- 
fits of  public  welfare  programs.   The  subcommittee  discovered  that  ad- 
ministrative costs  vary  from  two  to  twenty  percent  of  the  program  costs. 
Generally  only  a  small  fraction  of  actual  program  costs  are  used  on 
data  gathering,  research,  and  evaluation.   However^ much  of  this  data 
is  useless  for  good  evaluation. 

Some  discussion  focused  on  the  organization  of  program  evaluation 
within  agencies.   The  subcommittee  concluded  that  evaluations  should  be 
sponsored  by  an  office  attached  to  the  Office  of  the  Secretary  of  an 
agency  since  this  insures  better  access  to  decisionmaking. 

During  the  discussion,  Ms.  Townsend  elaborated  upon  a  point  she 
made  in  her  formal  presentation.   Committee  staff  found  it  necessary  to 
monitor  very  closely  the  work  of  their  outside  consultants,  especially 
social  science  researchers.   Even  though  researchers  are  on  contract  to 
answer  specific  policy  questions,  they  sometimes  orient  their  work  more 

to  meet  other  goals,  including  basic  research  and  teaching. 
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'POLICY  FORMATION  THROUGH  PROGRAM  EVALUATION  AND  SYSTEMS  ANALYSIS: 

A  CONGRESSIONAL  VIEW,"  JAMES  W.  GILTMIER,  SENATE  COMMITTEE  ON 

AGRICULTURE  AND  FORESTRY 


Ten  constituents  write,  "We  want  more  wilderness,"  and  a  congressio- 
nal eyebrow  lifts.   One  hundred  more  write  and  a  staff  project  is  initi- 
ated.  The  legislative  process  —  the  business  of  policy  formation  —  is 
in  action.   It  is  as  difficult  to  stop  as  it  is  to  start. 

Where  will  it  lead?  Who  knows? 

It's  simple  enough.   Find  out  what  all  sides  think;  run  it  by  the 
Administration  to  get  feelers  on  a  veto;  make  a  separate  check  with  the 
Office  of  Management  and  Budget  to  determine  what  that  secret  government 
might  think;  draft  a  final  bill;  run  it  past  a  subcommittee,  than  a  full 
committee  for  refinement;  pass  it  in  the  Senate;  adopt  the  conference 
report;  and  finally  send  it  to  the  White  House  for  the  President's  sig- 
nature. 

Throughout  this  process,  lawyer-lobbyists  on  both  sides  of  a  ques- 
tion, as  well  as  congressional  and  executive  department  lawyers  are  mov- 
ing the  commas  and  parentheses  around  in  mystical  ways.   Often  the 
process  becomes  so  cluttered  and  complicated  that  it  is  difficult  to  re- 
member that  "all  you  wanted  to  do  was  drain  the  swamp,"  or  create  a 
wilderness. 

The  product  of  all  this  is  that  which  is  politically  possible.   It 
is  not  always  good.   Often  that  product  is  an  overreaction  to  a  situ- 
ation, but  this  is  often  resolved  because  not  enough  funds  are  appropri- 
ated to  allow  the  overreaction  to  cause  too  much  damage. 

It's  a  sloppy,  unscientific  system  which  in  no  way  lends  itself  to 
a  systematic  approach.   But  it's  democratic,  and  that's  what  really  counts. 
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The  Founding  Fathers  didn't  promise  us  a  rose  garden,  but  they  did  offer 
the  people  a  voice  in  where  the  weeds  would  be  permitted  to  grow. 

Isn't  it  odd  that  the  compromise  and  bargaining  required  to  get  a 
measure  into  law  is  the  very  thing  that  many  Americans  detest  most  about 
our  political  system.   Yet  the  democracy  could  not  survive  without  the 
push  and  shove  of  interests  against  one  another  until  consensus  is  found. 
Sometimes  these  diverse  interests  are  regional,  sometimes  economic  and 
sometimes  social.   But  it's  important  in  adopting  legislation,  and  there- 
by setting  national  policies,  that  no  one's  ox  get  gored  too  much.   It's 
bad  enough  in  the  minds  of  many  Americans  that  there  must  be  a  Govern- 
ment at  all.   Therefore,  Government  excesses  could  tend  to  lead  to  a 
revolution. 

Nevertheless,  the  United  States  is  beginning  to  get  a  little  age  on 
it.   Congress  is  a  bit  more  mature  in  the  process,  and  there  is  a  large 
body  of  public  law  that  has  been  passed.   Many  of  the  members  of  our 
policy-making  arm  of  Government  are  wondering  if  law  is  not  passed  by 
listening  to  cases  being  overstated  on  either  side  of  issues,  without 
any  concern  for  whether  previous  laws  handle  the  situation  very  nicely, 
thank  you;  whether  the  executive  agencies  have  the  capacity  to  carry  new 
laws  out,  given  staff  and  funding  levels;  or  whether  the  good  intentions 
in  law  have  anything  to  do  with  the  delivery  of  services  to  the  public 
at  the  local  level.   For  that  matter,  there  are  questions  being  raised 
as  to  whether  Congress  is  adequately  staffed  to  know  what  happens  after 
legislation  is  adopted,  or  whether  it  is  equipped  to  determine  in  advance 
what  the  actual  effects  of  new  legislation  might  be. 
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A  sensitivity  to  all  of  this  led  Congress  to  adopt  the  Legislative 
Reorganization  Act  of  1970  —  which  in  itself  was  quite  a  compromise, 
and  according  to  some  elements  of  the  press,  not  much  of  a  reorganiza- 
tion at  all. 

Section  204  of  the  Act  directs  the  General  Accounting  Office,  either 
on  its  own  initiative  or  at  the  request  of  committees  of  the  Congress, 
to  make  studies  of  the  costs  and  benefits  of  existing  programs. 

Clearly  the  intent  of  this  is  to  see  if  we  have  any  old  laws  hang- 
ing around  that  aren't  much  good,  or  are  costing  the  taxpayers  more  than 
they  should  for  the  benefit  received,  and  the  process  should  also  be 
used  for  a  quick  informational  turnaround  on  whether  the  executive  agen- 
cies were  carrying  out  the  intention  of  Congress. 

Now  this  is  clearly  heady  stuff.   Implied  here  are  qualifiable  and 
quantifiable  measures  of  program  effectiveness  and  maybe  even  a  systems 
approach  to  policy  formation.   That  could  mean  computers,  information 
specialists  and  tons  of  paper  spewing  out  the  "who-shot-John"  of  govern- 
mental and  maybe  even  congressional  effectiveness.   Obviously  there  are 
implied  dangers  too  from  "business  as  usual"  bureaucrats  who  might  not 
particularly  care  about  what  the  wishes  of  Congress  might  be,  new  laws 
or  not. 

(The  GAO  authority  was  significantly  strengthened  last  year  by  en- 
actment of  the  Congressional  Budget  Impoundment  Control  Act.) 

It  would  only  be  fair  to  say  that  the  GAO  and  congressional  response 
to  the  Congressional  Reorganization  Act  was  timid.   Neither  side  wanted 
to  move  too  fast  for  fear  that  a  gigantic  structure  be  created  in 


227 


Government  that  might  amount  in  the  end  to  a  scandal-ridden  boondoggle 
that  delivered  too  much  information  to  be  usable  by  anyone,  while  cost- 
ing millions  of  dollars. 

Dr.  Ida  R.  Hoos  points  out  in  Datamation:   "Gargantuan  systems  that, 
with  the  ingestive  propensities  of  snakes,  have  subverted  and  swallowed 
up  the  very  organizations  they  were  purported  to  serve  are  the  outcome 
of  a  chain  of  apparently  innocuous  circumstances.   Electronic  data  pro- 
cessing introduced  in  the  mid-1950' s  to  hasten  the  paper  flow,  reduce 
clerical  costs,  and  increase  efficiency  in  information  handling  was  the 
first  step.   The  logic  was  clear,  economic  justification  unassailable, 
and  the  role  for  an  information  technologist  insured. 

"But  two  decades  and  two  computer  generations  later  the  promises 
are  still  illusory.   To  be  sure,  records  move  faster,  but  their  volume 
has  increased  exponentially.   In  a  Parkinsonian  fashion,  they  have  ex- 
panded to  fill  all  computer  time.   Clerical  costs  soar;  the  price  of 
processing  paper  is  rising  astronomically  but  invisibly,  as  any  accoun- 
tant charged  with  allocating  company  expenses  can  attest. "53/ 

Whoa!  Wait  a  minute.  Maybe  buttonholing  is  a  better  way  to  make 
policy  than  by  creating  "management  information  systems."  As  Senator 
Henry  Bellmon  of  Oklahoma  is  fond  of  saying,  "If  you  like  either  sausage 
or  law,  don't  ever  watch  either  of  them  being  made."  But  the  Government 
already  costs  more  than  $300  billion  a  year,  and  no  one  knows  where  the 
money  will  come  from  for  such  fixed  governmental  costs  as  civil  service 


53/  Hoos,  Ida  R.   Can  Systems  Analysis  Solve  Social  Problems?  Datamation, 
June  1974:83. 
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and  military  pensions  in  ten  or  fifteen  years.   Maybe  the  Congress  ought 
not  enter  into  something  that  could  be  a  very  large  ticket  item  without 
some  assurances  of  extensive  payoffs. 

On  the  other  hand,  Congress  is  timid  about  getting  into  something 
that  was  already  installed  by  former  Secretary  of  Defense  Robert  McNamara 
as  a  major  artery  in  the  operation  of  our  military  establishment. 

Even  though  McNamara' s  systems  proved  on  many  occasions  to  be  fatal- 
ly fraught  with  the  garbage-in,  gospel-out  syndrome,  Congressional  de- 
mands to  "prove"  why  the  defense  budget  should  be  so  high  required 
systematic  goal  setting  and  evaluation.   It's  now  a  big  business  at  the 
Pentagon.   It  is  growing  in  the  other  executive  agencies,  even  though 
one  official  in  the  office  of  Secretary  of  Defense  concluded  to  me  that 
while  program  evaluation  was  sophisticated  and  useful,  it  often  tended 
to  take  the  course  of  least  resistance,  supporting  previously  foregone 
conclusions. 

I  don't  know,  but  I  suspect  that  the  General  Accounting  Office  got 
its  early  strength  from  the  Al  Capone  case.   You'll  recall  that  even 
Elliot  Ness  couldn't  get  Al  into  Alcatraz  for  mutilating  the  bodies  and 
lives  of  his  enemies.   It  was  the  accountants  over  at  the  Internal  Rev- 
enue Service  who  nailed  him.   This  led,  I  believe,  to  the  generally  still- 
held  conclusion  that  today's  big-time  crooks  are  too  smart  to  be  caught 
with  guns.   It  takes  accountants.   Isn't  that  why  the  FBI's  list  of  ten 
most  wanted  generally  includes  only  psychopaths  and  poverty-stricken 
nobodies? 
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No  one  has  ever  been  able  to  explain  to  me  whether  GAO  is  an  arm  of 
Congress,  or  whether  it's  some  separate,  super-snoop  investigating  agency, 
held  in  check  only  by  its  conservative  and  thoughtful  Comptroller  General, 
Mr.  Staats.  While  it  serves  Congress  faithfully  with  its  carefully  worded, 
eternally  reviewed  reports,  more  than  70  percent  of  its  work  is  inspired 
from  within  the  agency.  Most  often  in  the  past,  the  GAO  approach  to  an 
investigation  was  the  auditor's  approach.   That  is,  are  those  bureaucrats 
out  there  doing  their  job  in  the  most  cost-effective  manner,  and  is  this 
program  really  necessary? 

That's  fine,  except  Congress  is  often  not  interested  in  whether  a 
program  is  cost-effective  in  the  most  literal  sense,  or  whether  a  pro- 
gram is  needed.  Government,  after  all  is  to  do  those  things  for  a  na- 
tion's citizens  which  they  want  done  but  which  they  cannot  do  for  them- 
selves. And  when  the  GAO  questions  the  efficacy  of  a  program,  the 
organization  is  questioning  the  wisdom  of  Congress  in  setting  up  some- 
thing in  the  first  place. 

It's  a  sensitive  business  as  GAO  recognized  in  its  first  attempts 
to  make  the  evaluation  process  more  consistent  with  the  Defense  Depart- 
ment approach.   Take  for  example  this  language  from  a  "Discussion  Paper 
on  Principles  and  Standards  for  Analysis  and  Evaluation"  drafted  by  GAO 
staff: 

Principles  and  standards  for  professional  practice  have 
existed  for  many  years  where  it  has  been  important  to  protect 
both  public  and  client  interests  by  the  use  of  generally  ac- 
cepted measures  of  competence.   Examples  are  the  practice  of 
public  accounting  where  the  financial  interests  of  investors 
are  at  stake;  and  the  practice  of  engineering  where  the  safe- 
ty of  the  public  is  at  stake.  More  recently,  there  is  a  gen- 
eral recognition  that  major  decisions  regarding  the  uses  and 
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amounts  of  funds  allocated  may  have  very  important  implica- 
tions for  the  general  public  as  well  as  the  direct  recipi- 
ents of  the  programs.   Since  these  decisions  have  been 
increasingly  influenced  by  program  evaluation  and  program 
analysis,  it  follows  that  there  is  a  need  for  generally 
accepted  principles  and  standards  to  govern  this  work. 

In  the  absence  of  such  principles  and  standards,  a 
number  of  problems  are  created.   For  example,  the  Urban 
Institute  reported  that:   'Two  Job  Corps  evaluations,  one 
using  'no  shows'  as  a  comparison  group,  and  the  other  using 
later  Job  Corps  enrolles  as  a  comparison  group,  reached 
widely  different  conclusions  on  the  value  of  the  Job  Corps; 
the  first  study  estimated  benefit-cost  ratios  of  4.5  to  5.0. 
This  led  one  observer  to  note:   'This  is  a  disturbing  situ- 
ation where  evaluation  relating  to  the  same  reality  may  vary 
from  submarginal  to  excellent,  depending  on  the  assumptions 
and  attitudes  of  the  analyst.' 

The  discussion  paper  then  goes  on  to  suggest  an  outline  of  the  sub- 
jects to  be  covered  in  an  orderly  set  of  principles  and  standards.   The 
major  outline  headings  are:   Purpose  of  Analysis  and  Evaluation;  Evalu- 
ation Methodology;  Reporting  Results;  and  Administration  of  Analysis  . 
and  Evaluation.   The  paper  makes  the  point  that  "studies  are  difficult 
to  use  for  congressional  decisions  unless  there  is  some  generally  ac- 
cepted criteria  for  judging  their  worth  and  unless  reports  contain  enough 
information  to  allow  such  judgments  tc  be  made." 

Dr.  Hoos  is  much  more  blunt.   She  claims  that  the  use  of  systems 
analysis  to  solve  social  problems  is  an  "opportunity  for  enormous  boon- 
doggling ...  by  the  very  ascription  of  rationality,  there  are  built-in 
assumptions  about  whose  conception  of  rationality  is  reflected.   Ratio- 
nality is  not  a  dye  vat  into  which  all  may  be  dipped  and  come  out  the 
same  color.   Rationality  is  neither  absolute  nor  constant,  but  depends 
on  the  frame  of  reference.   Rationality  knows  many  times,  places  and 
forms.   What  could  be  economically  rational  could  be  irrational  socially; 
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what  could  be  socially  rational  could  be  political  dynamite;  what  could 
be  politically  rational  could  be  totally  irrational  from  an  engineering 
point  of  view." 

To  cite  examples  of  what  can  happen,  Dr.  Hoose  points  to  Federal 
grants,  to  information  experts,  to  design  systems  which  diverted  people 
from  the  task  of  delivering  health  services  in  some  more  efficient  or 
effective  way.  "In  many  instances,"  she  says,  "it  costs  more  to  pro- 
cess information  about  a  patient  than  to  cure  him.  And  more  special- 
ists live  off  the  systems  set  up  to  conquer  certain  diseases  than  die 
from  those  diseases." 

Well,  at  least  there's  not  much  doubt  about  where  Dr.  Hoos  stands. 
She  thinks  we  are  wasting  a  lot  of  time  designing  systems,  models,  and 
games  when  we  ought  to  be  using  our  limited  resources  to  solve  problems 
with  more  obvious  solutions.   Her  point  about  whose  rationale  is  so 
relevant  to  the  politicians  that  they  have  bad  dreams  about  these  things 
at  night,  I  suspect. 

We  all  know  what  a  pilot  project  is;  it's  a  Government  activity 
without  enough  money  in  it  to  accomplish  very  much.  But  ideally  a  pilot 
project  is  an  attempt  at  something  new.   If,  after  evaluation  the  project 
pays  off,  then  Government  or  the  private  sector  at  some  level  presumably 
expands  the  idea  to  whatever  is  the  appropriate  universe.   An  example 
of  this  kind  of  approach  is  the  notion  by  the  Department  of  Housing  and 
Urban  Development  to  give  poor  people  money  so  that  they  can  purchase 
their  own  housing,  without  the  Government  having  to  go  through  the  busi- 
ness of  providing  the  more  familiar  types  of  housing  subsidies.   HUD 
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hired  a  number  of  contractors  to  do  elaborate  forms  of  evaluation  on  how 
this  might  work  out. 

However,  in  May,  Dr.  David  J.  Fitch, 54/  while  conceding  that  Con- 
gress needs  better  information  on  the  expected  effects  of  such  pilot 
programs,  added  two  more  problems  to  the  list  which  "make  it  difficult 
to  carry  out  and  evaluate  experimental  programs  .  .  .  : 

1.  The  agencies  which  have  responsibility  believe  that 
Congress  will  object  to  the  random  allocation  of  program  bene- 
fits needed  if  effects  of  the  programs  are  to  be  determined. 

2.  Those  within  an  agency  who  have  responsibility  for 
research  design  may  be  limited  by  their  own  competence  and 
are  almost  certainly  limited  by  those  above  them  who  feel  a 
need  to  'play  it  safe, '  and  therefore  will  not  support  good 
evaluation  research  design  because  of  anticipated  public  re- 
lations problems. 

My  thought  is  that  programs  with  regard  to  these  dif- 
ficulties could  be  made  through  the  establishment  of  a  grant 
operation  under  the  Senate  Government  Operations  Committee 
in  which  proposals  for  evaluation  research  from  the  agencies 
would  be  considered  by  a  staff  and  advisory  group,  and  funds 
awarded  to  support  good  research  —  the  philosophy  being  that 
the  more  Congress  demonstrates  it  wants  good  evaluation,  the 
better  the  evaluations  will  be. 

Ah,  but  Mr.  Fitch,  you  haven't  answered  Dr.  Hoos'  problem  about 
"whose  rationale." 

The  GAO  auditor  wants  to  be  isolated  so  that  he  can  draw  his  own 
"objective"  conclusions  —  in  other  words,  his  rationale.   Similarly, 
the  National  Science  Foundation's  RANN  support  of  evaluations  for  a 
series  of  public  concern  will  be  apart  from  Government.   Dr.  Selma  J. 
Mushkin  of  the  Public  Research  Laboratory  at  Georgetown  University 
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suggests,  "and  in  the  strength  of  its  isolation  from  Government,  it  may 
also  find  it  is  removed  from  the  realities  of  governments,  or  even  in 
its  isolation,  produce  an  environment  hostile  to  change." 

Dr.  Mushkin  shows  some  respect  for  the  evaluation  process  at  the 
Department  of  Defense,  without  endorsing  it  entirely: 

"It  is  a  model  that  yields  much  dialogue,  testing,  research,  re- 
assessment and  multiple  reviews  prior  to  application.   Yet  it  is  of 
Government,  produced  both  in  house  and  out  without  the  fanfare  that  has 
accompanied  some  civilian  sector  reports  ....   Surprisingly,  there 
appears  to  be  more  scientific  prodding  and  questioning  in  the  military 
system  than  is  currently  built  into  the  civilian  sector  evaluations." 

Dr.  Mushkin  points  out  that  in  the  military,  the  evaluation  pro- 
cesses built  in  at  all  levels  in  operations  and  budget  from  the  Secre- 
tary of  Defense  to  the  military  base,  whereas  multi-level  evaluation 
in  the  civilian  agencies  is  rare. 

She  says,  "evaluation  and  analysis  are  now  so  imbedded  into  the  prac- 
tices of  the  military  administrative  structure  that  I  think  it  would  be 
difficult  to  sort  out  the  analytical  resources  applied.   In  the  civil- 
ian sector,  each  evaluation  is  a  precious  thing,  or  nearly  so." 

Dr.  Mushkin  admits  having  no  regular  contact  with  the  military  eval- 
uative process.   But  it  would  only  be  fair  to  add  to  her  praise  that 
there  are  also  dangers  in  the  "everybody  gets  to  stir  and  season  the 
soup  theory."  Under  McNamara  particularly,  weapons  systems  tended  to 
become  more  cumbersome  and  complicated  than  necessary  because  each  com- 
mand had  something  it  wanted  added.   That's  how  you  get  individual  and 
vulnerable  war  machines  that  cost  millions  of  dollars  per  copy. 
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But  the  Importance  of  taking  more  than  today's  specific  problems 
into  account  is  made  evident  in  a  paper  Dr.  Mushkin  gave  in  1973  before 
the  American  Statistical  Association.   She  pointed  out  that  attempts 
toward  policy  analysis  in  the  health  field  cannot  be  limited  to  health 
programs. 

She  says,  "Benefit  assessments  underscore  the  commonality  of  pro- 
gram purpsoes.  Many  public  programs  involve  pricing  human  life,  programs 
that  are  not  directly  associated  with  health  care  and  that  do  not  bear 
the  label  of  health  programs.   The  most  familiar  example  is  the  highway 
construction  program,  where  lifesaving  costs  or  death  and  disability 
costs  must  be  equated  in  deciding  whether  or  not  to  construct  a  clover- 
leaf  or  add  another  highway  lane." 

It  gets  complicated,  and  Congress  knows  how  complicated  it  can  be 
as  evidenced  by  the  remarks  by  Congresswoman  Martha  Griffiths  at  a  hear- 
ing on  "Problems  in  Administration  of  Public  Welfare  Programs"  in  June  of 
1972: 

These  programs  affect  over  60  million  persons  and  cost 
$100  billion  a  year,  and  yet  for  years  we  have  examined  these 
programs  one  by  one.   But  both  the  Congress  and  the  executive 
branch  are  becoming  increasingly  aware  of  the  fact  that  we  can 
no  longer  afford  to  legislate  and  administer  the  programs  in  a 
piecemeal  fashion  when  so  many  of  the  programs  affect  the  same 
persons  ....   The  purpose  of  these  hearings  to  learn  more 
about  the  rules  and  regulations  under  which  programs  are  admin- 
istered at  the  local  and  state  levels,  and  the  actual  adminis- 
trative procedures  that  are  followed. 

Following  the  Congresswoman 's  opening  statement,  the  first  witness 

was  a  casework  aid,  who  said  immediately,  "I  think  the  most  important 

thing  that  I  mentioned  in  the  written  statement,  as  far  as  dealing  with 

our  clients  and  the  rules  and  regulations,  is  the  fact  that  the  grants 

are  so  low  in  relationship  to  the  need  that  our  clients  have." 
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Don't  groan.   That's  a  legitimate  statement.   As  a  matter  of  fact, 
I  don't  believe  Congress  would  give  program  evaluation  as  a  device  for 
policy  formation  a  second  thought  if  there  was  enough  money  to  go  around. 
But  the  members  watch  the  Federal  budget  closely,  and  each  year,  the 
fixed  costs  eat  up  more  and  more  spendable  dollars,  while  inflation  and 
interest  on  the  debt  chisel  away  at  the  other  end  of  things. 

Many  members  of  Congress  are  beginning  to  get  the  impression  that 
many  of  the  frustrations  of  the  people  with  the  Federal  establishment 
stem  from  the  fact  that  programs  often  do  not  operate  as  advertised  and 
even  if  they  do,  there  are  too  few  dollars  in  each  pot  to  solve  the  ori- 
ginal problem  for  all  of  the  prospective  users.  A  glaring  example  is 
the  food  stamp  program  which  in  just  a  few  years  has  soared  to  be  a  $5 
billion  program  while  at  the  same  time  advocates  of  the  program  complain 
only  about  50  percent  of  the  potential  users  are  being  served. 

As  a  result  of  situations  like  these,  Congress  wants  the  maximum 
program  bang  for  the  buck,  and  while  it  wants  to  know  which  programs  work 
and  which  ones  don't,   the  members  clearly  aren't  too  happy  when  the  cat- 
egorical cripples  are  exposed  in  the  daily  press  before  there  is  time  to 
make  necessary  orthopedic  repairs. 

The  Budget  Control  and  Impoundment  Act  adopted  this  year  will  force 
many  changes,  the  most  basic  being  that  under  ideal  operation,  the  De- 
fense Department  will  have  to  line  up  its  priorities  alongside  those  of 
the  Departments  of  Health,  Education  and  Welfare,  and  Agriculture  so  that 
Congress  can  decide  which  things  are  most  important  from  a  national  stand- 
point to  pour  the  dollars  into.   It  is  not  too  difficult  to  foresee  one 
of  two  things  happening: 
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Either  the  executive  agencies  will  begin  to  do  their  own  program 
evaluation  to  provide  their  authorizing  committees  with  the  ammunition 
to  defend  their  operations  before  the  budget  committees,  or  all  deci- 
sions will  be  made  at  the  White  House  and  cabinet  officers  will  be  rele- 
gated to  housekeeping  chores  like  second  lieutenants  at  the  Pentagon. 
If  the  latter  selection  is  chosen  by  an  Administration,  there  will  un- 
doubtedly be  many  head-to-head  confrontations  between  Congress  and  the 
White  House.   The  winners  in  cases  like  this  will  be  those  programs  with 
the  strongest  lobbies  backing  them  up,  or  the  annual  glamour  programs 
which  the  press  enhances  until  it  gets  bored  with  them  —  you  know  what 
I  mean:   last  year  the  environmental  crisis,  and  this  year  the  energy 
crisis. 

There  may  not  be  much  room  for  appropriations  by  merit  if  this 
course  is  taken,  and  if  recent  history  is  a  measure  to  judge  by,  pro- 
grams for  rural  people  will  probably  suffer  at  the  hands  of  an  urban 
dominated  House  anxious  to  take  care  of  its  majority. 

Nevertheless  —  the  new  budget  control  act  aside,  Congress  wants 
to  know  which  programs  work  and  which  programs  are  not  providing  good 
public  service  given  the  dollars  spent,  and  even  though  my  instincts 
tell  me  to  agree  with  Dr.  Hoos  when  she  says,  "Systematic  and  analytic 
review  of  the  systems  techniques  would  reveal  that  they  are  not  appro- 
priate to  the  myriad  of  social  tasks  to  which  they  are  being  applied," 
I  continue  to  be  left  with  the  frustration  that  I  do  not  really  know 
what  is  going  on  out  "on  the  ground,"  as  the  foresters  say,  and  there- 
fore I  often  find  it  difficult  to  provide  the  Senators  I  serve  with  the 
reliable  information  they  deserve.   I  know  further  that  one  of  the  reasons 
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I  don't  know  what  is  happening  is  because  of  intentional  obfuscation  by 
Federal  agencies  in  many  cases.   Like  the  Job  Corps  survey  mentioned 
earlier,  they  can  make  the  numbers  say  anything  they  want  —  or  anything 
that  they  think  Congress  wants  to  hear. 

When  GAO  came  to  see  the  staff  of  the  Committee  on  Agriculture  and 
Forestry  following  enactment  of  the  Congressional  reorganization  bill, 
there  was  polite  exchange  of  detailed  letters  between  the  Chairman, 
Senator  Talmadge,  and  the  Comptroller  General.   These  sallies  were  fol- 
lowed by  a  round  of  meetings  while  the  GAO  tried  to  find  out  whether  we 
wanted  to  dump  ten  computer  years  of  work  on  them,  and  we  tried  to  learn 
whether  it  would  take  a  year  to  get  a  turnaround  answer  on  a  problem 
from  GAO,  as  we  felt  was  usually  the  case. 

These  conversations  went  on  for  hours,  gradually  becoming  more  and 
more  refined  as  the  GAO  staff  began  to  understand  our  point  of  view  on 
agriculture,  rural  development  and  forestry.   It  was  only  then  that  we 
could  get  down  to  the  business  of  problem  solving. 

So  you  see,  in  this  case  it  was  our  rationale. 

Since  these  early  meetings,  the  GAO  has  performed  a  number  of  worth- 
while services  for  the  Committee  —  most  of  them  rather  homely  tasks  that 
have  not  required  management  information  systems  or  computers.   The  GAO 
systems  people  worked  with  us  in  a  somewhat  more  informal  basis  than  had 
been  their  usual  procedure,  providing  on-the-spot  advice  that  was  of  more 
value  than  the  written  GAO  report  which  gees  through  a  final  review  pro- 
cess that  takes  two  months  or  more. 
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As  an  aside  here,  it  should  be  noted  that  the  seven  or  more  review 
steps  plus  executive  agency  review  of  a  report  is  a  scholarly  way,  and 
auditor's  way  of  going  about  a  study.   Often,  things  move  too  quickly 
on  Capitol  Hill  for  that  kind  of  work  to  be  of  much  value.   For  instance, 
with  the  right  kinds  of  pressures  behind  it,  the  committee  can  receive 
a  bill,  hold  hearings,  and  have  it  reported  to  the  floor  in  two  weeks. 
But  obviously  there  are  long-term  studies  which  are  worth  doing.   The 
committee  has  a  huge  job  underway  right  now.   It  involves  a  comprehen- 
sive review  of  the  personnel  of  the  Farmers  Home  Administration  to  mea- 
sure the  competence,  number,  and  job  attitudes  of  the  personnel  matched 
up  against  the  programs  which  Congress  has  given  the  agency  to  accom- 
plish. 

This  study  is  interesting  because  Senator  Dick  Clark,  who  asked  for 
the  study,  declined  to  call  it  an  investigation.   Further,  Clark  has  en- 
listed and  received  the  cooperation  of  the  Administrator  of  Farmers  Home 
Administration  in  assisting  GAO  with  the  work.   Thus  far,  GAO  and  Farmers 
Home  have  been  working  at  the  job  in  an  admirable  manner. 

It  is  our  hope  that  the  completed  study  will  draw  rather  broad  sets 
of  conclusions  upon  which  the  committee  can  base  future  decisions  con- 
cerning the  future  of  the  agency,  and  those  decisions  can  be  made  in 
such  a  way  that  in  negotiating  any  changes  in  law  or  administrative  regu- 
lations with  the  Farmers  Home  Administration,  both  the  committee  and  FmHA 
will  be  working  from  the  same  sets  of  numbers  and  assumptions. 

Hopefully  the  results  of  this  study  will  provide  a  vital  tool  to 
aid  in  policy  formation. 
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Also  with  the  help  of  the  GAO,  the  committee,  under  the  leadership 
of  Senator  Humphrey,  has  had  enacted  a  forest  and  rangeland  management 
bill  which  is  designed  to  provide  us  with  a  new  systematic  approach  to 
decision-making  on  the  National  Forest  System.   We  believe  that  this 
bill  could  serve  as  a  model  for  future  legislation  in  other  fields. 

For  the  past  several  years,  the  air  of  Washington  has  been  polluted 
with  a  mean  little  debate  over  the  management  of  the  National  Forests. 
The  Forest  Service,  surrounded  by  assaults  from  conservation  organiza- 
tions and  the  timber  industry  was  under  a  constant  legal  and  verbal 
cloud.   The  agency  was  losing  the  confidence  of  the  Office  of  Management 
and  Budget  to  cope  with  its  problems,  and  there  is  even  evidence  that 
there  was  a  dispute  within  OMB  on  how  the  National  Forests  should  be 
managed.   Because  of  inadequate  appropriations,  millions  of  acres  of  the 
forests  needed  reforestation  or  stand  improvement.   Because  of  drastic 
personnel  cuts  by  the  Administration  —  even  in  the  face  of  orders  to 
cut  more  trees  —  the  Forest  Service  was  incapable  of  managing  any  of 
its  other  responsibilities  as  well  as  it  knows  how. 

The  problem  as  Senator  Humphrey  saw  it,  was  to  cut  the  trees  the 
Nation  had  to  have  (lumber  prices  were  skyrocketing)  and  still  provide 
for  wilderness,  other  forms  of  public  recreation,  protect  watersheds, 
wildlife  and  generally  maintain  a  sound  environmental  balance  in  the 
forests. 

Once  again,  skull  sessions  got  the  job  done.   The  staff  of  the  Com- 
mittee sat  down  with  representatives  of  a  number  of  conservation 
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organizations.   At  the  same  time,  and  the  same  table,  we  had  representa- 
tives of  the  industry  and  occasionally  personnel  from  the  Forest  Service. 

The  result  of  those  meetings  was  not  a  new  policy  for  the  National 
Forest  System  but  a  device  by  which  policy  can  be  set. 

Briefly,  the  bill  provides  that  periodic  assessments  be  made  of  all 
of  the  renewable  resources  in  and  on  the  National  Forest  System.   Using 
this  assessment,  the  President  is  to  develop  a  Program  of  Work  for  ten- 
year  increments  that  is  amendable  by  Congress.   After  the  program  is 
adopted,  the  President  is  required  each  year  to  request  such  sums  as  are 
necessary  to  meet  the  goals  of  the  program,  or  if  he  does  not  ask  for 
the  money,  he  must  explain  why. 

Finally  —  and  this  is  where  GAO  helped  us  out  —  the  bill  provides 

for  program  evaluation  which  will  aid  the  Congress  in  determining  whether 

the  Forest  Service  is  getting  the  job  done.   The  evaluation  language 

reads  like  this : 

For  the  purpose  of  providing  information  that  will  aid 
Congress  in  its  oversight  responsibilities  and  approve  the 
accountability  of  agency  expenditures  and  activities,  the 
Secretary  (of  Agriculture)  shall  prepare  an  annual  report 
which  evaluates  the  compenent  elements  of  the  program  re- 
quired to  be  prepared  .  .  .  which  shall  be  furnished  to  Con- 
gress at  the  time  of  submission  of  the  annual  fiscal  budget, 
commencing  with  the  third  fiscal  year  after  the  enactment 
of  this  Act. 

These  annual  evaluation  reports  shall  set  forth  pro- 
gress in  implementing  the  program  required  to  be  prepared 
by  Section  3  of  this  Act  together  with  accomplishments  of 
this  program  as  they  relate  to  the  objectives  of  the  as- 
sessment.  Objectives  should  be  set  forth  in  qualitative  and 
quantitative  terms,  and  accomplishments  should  be  reported 
accordingly.   The  report  shall  contain  appropriate  measure- 
ments of  pertinent  costs  and  benefits.   The  evaluation  shall 
assess  the  balance  between  economic  factors  and  environmen- 
tal quality  factors.   Program  benefits  shall  include,  but 
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not  be  limited  to,  environmental  quality  factors,  such  as 
esthetics,  public  access,  wilflife  habitat,  recreational  and 
wilderness  use,  and  economic  factors  such  as  the  excess  of 
cost  savings  over  the  value  of  foregone  benefits  and  the 
rate  of  return  on  renewable  resources. 

The  reports  shall  indicate  plans  for  implementing  cor- 
rective action  and  recommendations  for  new  legislation, 
where  warranted. 

The  reports  shall  be  structured  for  Congress  in  concise 
summary  form  with  necessary  detailed  date  in  appendices. 

I  don't  know  what  that  language  does  for  my  reader,  but  for  me  it 
sings. 

Similar  evaluative  efforts  by  GAO  have  assisted  the  committee  in 
its  extremely  detailed  oversight  hearings  in  the  field  of  rural  devel- 
opment, an  area  in  which  the  Nixon  administration  did  not  choose  to  be 
responsive.   An  unfortunate  result  of  this  was  that  our  detailed,  sys- 
tematic questioning  before,  during,  and  after  oversight  hearings  kept 
the  small  staff  of  the  Rural  Development  Service  in  USDA  totally  tied 
up  for  weeks,  with  no  time  to  d£  any  rural  development,  even  if  it  had 
been  permitted  to  do  so  by  the  administration. 

However,  it  would  be  well  to  add  that  Congress  now  has  the  Admin- 
istration's attention  in  this  subject  area,  despite  the  Executive's  de- 
sire for  the  whole  business  to  go  away. 

As  Dr.  Arthur  Miller,  the  constitutional  scholar  at  George  Washing- 
ton University  points  out,  ours  is  a  Government  that  provides  for  a 
strong  Executive.  As  he  says,  no  other  major  nation  puts  so  much  power 
into  the  hands  of  one  man.  In  the  past,  Congress  has  added  to  those  pow- 
ers by  receding  from  its  own  prerogatives.   But  I  believe  those  days  are 
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over.   Congress  has  taken  a  number  of  steps  such  as  passage  of  the  Bud- 
get and  Impoundment  Act  to  regain  some  control  over  policy  and  spending. 
There's  a  new  day  coming,  but  it  will  not  arrive  next  week,  because 
change  in  a  democracy  is  necessarily  slow  and  evolutionary  (unless  the 
protein  of  revolution  of  some  kind  is  combined  in  the  admixture) .   But 
the  Congress  is  leery  of  "experts"  who  have  often  led  the  member  down 
the  garden  path,  and  it  is  comfortable  with  the  voice  of  home  folks, 
who  are  not  experts. 

By  and  large,  the  lawyers,  economists  and  generalists  who  are 
the  staff  of  Congress  are  more  aware  that  the  term  ought  to  be  "systems 
analyses"  or  "system  analysis"  (as  opposed  to  systems  analysis)  than 
they  are  of  what  management  information  systems  can  do  to  help  them 
serve  their  masters.   Further  we  are  very  nervous  about  using  machines 
to  put  people  problems  into  neat  cubbyholes.   But  one  thing  is  certain  - 
Congress  and  its  staff  have  one  basic  complaint: 

Somehow  we've  got  to  find  out  what's  happening  in  those 
damn  agencies  so  we  can  make  some  decent  decisions  around  here 
for  a  change. 

That  basic  frustration  will  lead  Congress  to  do  something,  but 

again,  since  there  are  535  individuals  on  board,  God  only  knows  what 

that  something  will  be,  and  it  may  be  535  somethings  which  may  or  may 

not  be  subject  to  systematic  evaluation. 
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DISCUSSION 

During  the  course  of  his  research  evaluating  programs  of  the 
Department  of  Agriculture,  Mr.  Giltmier  discovered  that  the  agency 
refuses  to  release  some  evaluation  reports  to  the  Congress.   In  this 
same  respect  because  of  a  lack  of  cooperation  from  the  Department  of 
Agriculture,  it  took,  the  General  Accounting  Office  eight  months  to 
compile  a  list  for  the  committee  of  evaluation  reports  done  by  or 
for  the  Department  of  Agriculture. 

Mr.  Giltmier  noted  also  that  sometimes  the  General  Accounting 
Office's  assistance  is  inadequate  for  the  committee's  legislative 
oversight  needs.   One  clear  cause  for  this  inadequacy  is  the  "long 
turn-around  time"  for  much  GAO  work.   The  Office  employs  a  seven- 
step  review  process.   As  a  result  a  year  may  elapse  between  the  com- 
mittee's request  for  a  project,  its  completion,  the  seven-  step 
review  and  delivery.   The  committee  may  find  these  time-delayed  re- 
ports inadequate  because  members  are  no  longer  interested  in  the  is- 
sue.  It  would  be  helpful  if  the  GAO  could  provide  evaluation  in- 
formation in  a  shorter  time. 

Recently  the  General  Accounting  Office  instituted  two  reporting 
procedures  which  circumvent  the  need  for  long  reviews,  thereby  pro- 
viding the  Congress  with  information  on  a  more  timely  basis.   These 
consist  of  (1)  presenting  requesting  committees  with  oral  briefings, 
or  (2)  transmitting  informally  reviewed  draft  reports,  which 
committees  sometimes  use  as  final  reports. 

Mr.  Giltmier  noted  another  potential  problem:  that  the  objec- 
tives of  many  GAO  studies  do  not  coincide  with  legislative  infor- 
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mation  and  oversight  needs.   It  was  noted  that  seventy  percent  of 
GAO  studies  are  self-initiated.   However,  it  was  also  remarked  that 
many  of  'the  GAO  studies  are  done  to  fulfill  statutory  requirements 
made  by  the  Congress. 

The  point  was  also  made  that  sometimes  legislative  actions 
themselves  hamper  effective  evaluation.   Several  factors  contribute 
to  this  dilemma.   First,  many  different  laws  are  enacted  relating 
to  similar  types  of  programs  and  program  delivery.   Sometimes  these 
overlap  or  are  duplicative  and  sometimes  they  are  contradictory. 
This  happens  especially  in  the  agricultural  area.   The  suggestion 
was  made  that  before  drafting  new  legislation,  staff  should  spend 
more  time  determining  if  there  already  are  laws  relating  to  the  is- 
sue, or  if  existing  statutes  merely  require  modification. 

The  second  problem,  mentioned  repeatedly  throughout  the  seminar^ 
is  the  fact  that  legislative  statements  of  program  goals  or  objectives 
are  imprecise  or  vague.   This  makes  it  difficult  for  evaluation  re- 
searchers, including  those  in  the  GAO,  to  develop  yardsticks  of  pro- 
gram goals  against  which  to  measure  actual  accomplishments.   It  is 
difficult  to  reconcile  this  problem  because  of  the  need  for  political 
compromise  in  writing  and  passing  legislation. 
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DAY  III:  EXECUTIVE  BRANCH  EVALUATIONS  AND  THE  PROCESSES 
OF  EVALUATION:  SUMMARY  OF  PROCCEDINGS 


A.  Activities  Underway  In  the  Office  of  Management  and  Budget  to  Improve 
Evaluation  Activities 

Clifford  W.  Graves  was  the  lead-off  speaker  for  the  third  day 
of  the  seminar,  May  19.   He  began  his  presentation  by  tracing  the  evolu- 
tion of  the  Bureau  of  the  Budget's  and  the  Office  of  Management  and 
Budget's  concepts  of  evaluation.   At  first  the  Bureau  of  the  Budget  viewed 
evaluation  exclusively  in  terms  of  efficiency.   Subsequent  attempts  by 
the  BOB  and  the  OMB  to  view  evaluation  in  terms  of  program  performance 
and  accomplishments,  using  the  tools  of  the  Planning,  Programing,  Budget- 
ing System  and  Management  by  Objectives,  did  little  to  meet  decision- 
making needs.   However,  these  efforts  left  an  important  legacy  for  the 
present,  since  they  influence  the  mode  of  thinking  of  staff  of  OMB,  GAO, 
and  Federal  agencies,  and  they  provide  a  pool  of  people  experienced  in 
thinking  in  terms  of  program  performance  and  output  measurements. 

The  OMB  has  conducted  several  assessments  of  the  quality  and  use  of 
Federally  funded  evaluation  studies.   The  assessments,  according  to  Mr. 
Graves,  indicate  that  most  of  these  evaluation  efforts  were  not  timely 
enough,  inaccurate,  and  useless  or  irrelevant  for  decisionmaking  needs. 
The  OMB  is  now  pursuing  a  vigorous  effort  to  improve  the  conduct  and 
use  of  Federally  funded  evaluations.   The  OMB  Office  emphasizes  that  evaluation 
should  not  be  an  academic  exercise,  but  rather  should  be  an  integrali 
on-going  part  of  Federal  agency  operations,  effectively  integrated  into 
management,  program  planning,  and  preparation  of  budget  justifications. 
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The  OMB  provides  technical  assistance  and  policy  directives  on 
evaluation  to  Federal  agencies.   In  the  process,  however ,  the  agency 
stresses  that  Federal  agencies  themselves,  and  not  the  OMB  or  the  Con- 
gress, should  be  the  focal  point  for  improving  the  conduct  and  use  of 
evaluations. 

OMB  efforts  to  strengthen  agency  evaluation  capabilities  consist  of 
encouraging  agencies  to  use  a  "decision-driven"  model  for  evaluation. 
This  requires  agencies  to  gauge  where  evaluation  will  be  used  in  decision- 
making before  funding  evaluation  research.   At  the  same  time  the  OMB  is 
encouraging  the  Congress  to  include  appropriate  program  evaluation  re- 
quirements into  legislation  authorizing  new  programs. 

During  1974  the  OMB  prepared  an  inventory  of  Federal  agency  evaluation 
research  activities  and  offices.   It  is  conducting  a  joint  project  this  year 
with  the  General  Accounting  Office  to  update  this  study  and  to  gather 
more  information  about  the  funding  of  evaluation  research.   The  OMB 
recently  published  a  guide  for  managing  evaluations  in  Federal  agencies 
and  has  plans  for  the  following  publications:   an  assesment  of  the  use 
of  agency  evaluations,  an  assessment  of  evaluation  training  needs,  and 
a  technical  manual  to  assist  agencies  in  structuring  evaluation  manage- 
ment systems. 

B.  Activities  Underway  in  the  General  Accounting  Office  to  Improve 
Evaluation  and  Evaluation  Research 

The  Congressional  Budget  and  Impoundment  Control  Act  of  1974,  P.L. 

93-344,  gave  the  General  Accounting  Office  several  responsibilities  to 

assist  the  Congress  in  improving  program  evaluation.   Among  these  are  the 

responsibility  for  providing  assistance  to  requesting  committees  on  evaluation 

methodology,  helping  committees  to  develop  guidelines  for  legislative 
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language  for  evaluation,  developing  a  capability  for  synthesizing  evalua- 
tion reports,  and  cooperating  with  the  Office  of  Management  and  Budget 
and  relevant  committees  in  developing  program-related  information  systems. 

Keith  Marvin  of  the  General  Accounting  Office  provided  details 
about  activities  the  agency  is  taking  to  implement  these  responsibilities. 
They  include: 

-  development  for  each  agency  of  an  inventory  of  evaluations  completed, 
in  process,  or  planned,  including  a  short  description  of  the  agency's 

total  evaluation  plan; 

-  development  of  an  inventory  for  each  agency,  giving  the  title  of 

an  evaluation  report,  the  title  of  the  program  evaluated,  the  date  the 
legislation  was  authorized,  and  the  date  of  the  evaluation  report; 

-  preparation  of  abstracts  of  each  evaluation  report,  identifying 
the  source  or  performer;  and 

-  development  of  criteria  and  standards  for  congressional  committees 
doing  evaluations,  to  assist  them  in  making  tradeoffs  between  different 
types  of  evaluation  strategies  in  relation  to  specific  evaluation  related 
decisionmaking  needs. 

C.   The  Preparation  of  a  Draft  Manual  on  Evaluation  for  Legislative  Users 
Harrison  Fox  is  Research  Director  of  the  staff  of  Senator  Bill  Brock. 
William  Foskett  has  served  as  a  part-time  consultant  to  the  Senator. 
These  two  staff  members  have  made  important  contributions  to  improving 
legislative  staff  capabilities  for  evaluation.   For  instance,  under 
Senator  Brock's  authorization,  Dr.  Fox  has  held  a  monthly  seminar  for 
staff,  bringing  together  legislative  users  of  evaluation,  executive  branch 
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program  managers  and  users  of  evaluation,  and  evaluation 
performers  to  discuss  case  studies,  general  methodology,  and  ways  of 
enhancing  the  use  of  evaluation  research  in  decisionmaking.   Also,  Drs. 
Fox  and  Foskett  are  co-authors  of  a  draft  Program  Evaluation  Manual  for 
Legislators  and  Legislative  Staffs.   The  manual  gives  general  principles 
about  evaluation  in  terms  relevant  to  legislative  staff  who  have  responsi- 
bility for  incorporating  evaluation  requirements  into  legislation. 

Dr.  Fox  began  the  joint  presentation  by  describing  the  importance 
of  evaluation  as  a  tool  of  oversight.   But  he  also  noted  that  this  tech- 
nique is  only  one  of  several  essential  oversight  tools  the  Congress  uses. 
He  gave  the  following  typology: 

Type  of  Oversight  Methods  Which  Can  be  Used 

Technique 


Legislative 


Hearings  and  meetings 
Congressional  veto 
Statutory  controls 
Nonstatutory  controls 
Reporting  requirements 


Fiscal  and  Analytical 


The  appropriations  process 
Control  through  audit 
Program  evaluation 
Achieving  priorities 


Investigative 


Investigations 
Constituents  Problems 


Authoritative 


Impeachment 
Confirmation 
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Dr.  Foskett's  presentation  consisted  of  a  short  description  of  the 
manual  and  its  uses  on  the  basis  of  his  several  years  of  experience 
conducting  evaluations  in  education  and  developmental  psychology,  he  also 
commented  on  the  conduct  of  evaluation. 

Like  Dr.  Townsend,  Dr.  Foskett  noted  that  legislative  staff  must 
work  very  carefully  in  developing  legislative  language  for  evaluation. 
Staff  do  not  necessarily  have  to  be  concerned  with  the  specifics  of  an 
evaluation  research  design,  but  it  is  essential  that  they  be  familiar 
with: 

-  the  need  to  formulate  correct  legislative  statements  of  program 
goals,  capable  of  serving  as  yardsticks  or  achievements  against  which 
programs  can  be  evaluated; 

-  general  methods  of  experimental,  quasi-experimental,  and  non- 
experimental  evaluation  research  designs,  including  their  funding  and 
time  requirements,  and  the  differences  among  types  of  evaluation  in 
relation  to  legislative  needs,  in  order  to  formulate  legislative  re- 
quirements for  the  right  type  of  evaluation  design  in  relation  to  the 
questions  which  need  to  be  answered;  and 

-  the  need  for  close  communication  between  user  and  doer  of  evalua- 
tion to  encourage  the  generation  of  information  most  relevant  to  decision- 
making needs. 

Legislative  users  must  also  be  aware  of  the  political  constraints 
on  evaluation,  that  is, be  able  to  identify  evaluation  priorities  in 
relation  to  political  expediency,  and  to  know  when  and  why  agencies 
might  choose  not  to  evaluate,  or  not  to  use  the  evaluation  information 
which  has  been  generated. 
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As  another  important  point,  legislative  users  who  have  a  responsibility 
for  writing  legislative  language  for  programs  requiring  evaluation 
should  be  aware  that  some  types  of  quasi-experimental  and  experimental 
impact  evaluations  necessitate  expensive  longitudinal  data  collection 
procedures. 

Legislative  users  also  must  make  tradeoffs  on  another  issue,  that 
is,  can  existing  evaluations  of  these  programs  or  syntheses  of  evaluations 
of  the  same  or  related  programs  serve  the  evaluation- related  decision- 
making needs  of  the  Congress. 

Utilizing  a  framework  developed  for  the  manual,  Dr.  Foskett  then 
outlined  the  types  of  evaluation  techniques  and  tools  which  could  help 
generate  information  for  twenty  different  types  of  legislative  over- 
sight questions: 


251 


■tutkiu  i*  raiquu  aica  ht  u  to  ma  onuicrr  qnrrion  uits  in  is-ituum  i 


mmu  m  ...»i.«i  »«■ 


.,..    t.1^1    Ijcknl.* 


I.    UUiwTfM    rolrl.v.1;    pilot    tMU0«. 


Pupplp    rmJKi!      To   proriao    lnc« 

«'»f  «i  tYtfiT  f<mi 


I.  Miicy  iuImii; 


pilot    tutU|;    pi—  ll  f  II 111    polity   •Ml/cl* 


»ll»  —  *4»»c7*t  »1— .   orogroa   I 


U    lo.lil.tl.ro    htitftrj) 


if—"1":  ^  *-»" 


oooaot  of  mlookmt;;   unlilutM  of   ro*al»»..   f..  rn.la.iio. 


flui  u-17.1. 


pro»rW.».l««tlo«  4**1...    ojoji 


I"..    tKo   tool.  or  nooo. 


;l.l«cloa  orl.lnollr   .date 


•ftovoh   Co   iolvo    tho    pto.1*. 


M.l».ll,    .«,.....    ., 


D.  Issues  Relating  to  the  Methodology  of  Evaluation 

From  his  vantage  point  as  a  social  scientist  and  the  president  of  a 
large  consulting  firm  which  does  a  considerable  number  of  evaluations  for 
the  government,     Clark  Abt  presented  a  comprehensive  statement  on  the 
state  of  the  art  of  the  methodology  of  evaluation  and  on  issues  surrounding 
the  use  of  evaluation. 
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Although  he  devoted  a  considerable  portion  of  his  paper  to  the 
methodological  limitations  of  evaluation,  Dr.  Abt  noted  that  some  kinds 
of  evaluations  are  more  methodologically  sound  than  others.   For  instance, 
it  is  easier  to  do  cost-ef fectiveaess  evaluation  than  cost-benefit  evalua- 
tion.  In  cost-effectiveness  evaluation  "some  criterion  of  effectiveness 
is  set,  such  as  advance  in  reading  scores  in  an  educational  program,  and 

the  amount  of  advance  per  unit  of  cost  is  the  cost-effectiveness It's 

basically  a  measure  of  the  desired  program  output  over  cost  input." 

"Benefit-cost  is  harder  to  compute  because  here  we  are  computing 
benefits  and  costs  in  commensurate,  usually  dollar  terms.   We're  trying 
to  estimate  what  amounts  to  computation  of  return  on  investment,  so  that 
we  can  compare  different  programs.   It  is  hard  to  compare  years  of  read- 
ing achievement  per  dollar,  with  percentage  reduction  in  cancer  fatali- 
ties per  dollar  two  completely  different  programs  in  education  and 

health.   If  you  have  benefit-cost  ratios  for  both,  or  return  on  fungible 
resources  invested,  you  can  compare  those  and  when  you  can  compare  across 
programs  ideally,  you  can  optimize  the  whole  protfolio  of  government 
investments  by  what  the  economists  call  marginal  utilities. 

However,  the  major  obstacle  to  effective  evaluation,  according  to 
Dr.  Abt,  is  the  fact  that  it  is  not  used  by  decisionmakers.   Several 
factors  account  for  this  situation.   One  is  that  some  decisionmakers 
expect  more  of  evaluation  than  it  can  deliver.   For  example  ,  he  noted: 
"What  we  cannot  do  with  program  evaluation  is  create  new  programs,  make 
decisions,  make  the  assumption  of  rationality  come  true,  and  get  users 
or  consumers  of  program  evaluations  to  ask  the  right  questions.   We  also 
cannot  make  value  judgments  that  have  any  credibility  where  these  are 
decisive  program  decisions." 
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He  also  noted  that  communications  gaps  besiege  both  doers 
and  users.   Legislation  does  not  always  specify  properly  the  objectives 
and  potential  decisionmaking  needs  of  the  evaluation  study.   This  situa- 
tion could  be  improved  if  evaluation  were  conceived  of  as  a  "decision- 
driven"  technique  to  be  used  in  specific  legislative  oversight  functions, 
such  as  budgeting,  program  formulation,  or  management.   Related  to  this 
point  is  the  abrogation  of  responsibility  by  the  doers  of  evaluation, 
who  are  not  sensitive  to  the  political  needs  or  connotation  of  evaluation 
information  and  make  little  effort  to  aid  decisionmakers  in  assessing 
the  policy  implications  of  evaluation. 

Abt  also  stated  that  the  users  of  evaluation  are  not  always 
aware  of  the  possibilities  and  limitations  of  evaluation  research.   As  a 
result  users  frequently  do  not  ask  the  right  questions  of  evaluation  or 
do  not  receive  adequate  payoff  for  the  dollars  invested  in  evaluation 
research.      Abt  noted,  for  instance,  that  never  once  in  his  twenty 
year  career  as  an  evaluation  researcher  had  legislative  users  of  evalua- 
tion consulted  with  him  regarding  the  objectives,  methodology,  or  possible 
use  of  an  evaluation  study. 

Dr.  Abt  emphasized  that  while  the  technology  of  evaluation  research 
had  improved  considerably  in  recent  years  there  have  been  very  few 
theoretical  breakthroughs  in  evaluation  research  methodology  or  concepts. 
Both  evaluators  and  decisionmakers  have  a  responsibility  to  improve  this 
situation.   They  can  do  this  by: 

-  requiring  evaluations  only  of  the  most  important  priority  programs, 
not  of  every  program  legislated; 

-  doing  cross-cutting  evaluations,  which  look  at  the  objectives, 
effectiveness  and  impacts  of  programs  with  similar  goals,  authorized  by 
different  statutes,  and  administered  by  different  Federal  or  state  agencie 
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-  requiring  Federal  evaluation  managers  to  write  better  RFPs  and 
modifying  procurement  regulations  so  that  technically  non-competitive 
firms  do  not  waste  inordinate  amounts  of  time  and  money  writing  pro- 
posals, and 

-  sensitizing  decisionmakers  to  make  decisions  (and  to  accept  the 
consequence^  about  when  "quickie"  evaluations  are  required  and  when 
longer,  experimental  evaluations  with  randomized  test  and  control  groups 
are  needed. 

Among  the  other  weaknesses  Dr.  Abt  identified  were: 

-  inadequate  Federal  support  for  training  evaluatbrs  in  the  use 
of  statistical  methods  of  analysis; 

-  insensitivity  to  the  selection  of  indicators  of  effectiveness 
and  absence  of  well-formulated  relationships  between  measures  of  effec- 
tiveness and  possible  policy  options; 

-  inadequate  review  by  evaluators  of  the  literature  which  already 
exists  on  evaluation;  and 

-  insufficient  awareness  by  both  evaluatorB  and  decisionmakers 
about  the  possible  effects  of  an  evaluation  or  an  experiment  on  the 
participants  of  the  program. 

E.  Evaluation  in  the  Office  of  Education 

Dr.  John  Evans,  Assistant  Commissioner  of  Education  in  the  Office  of 
Education,  described  issues  relating  generally  to     federal  procure- 
ment and  use  of  evaluation,  using  the  Office  of  Education  as  an  illustra- 
tion. 

He  began  by  describing  several  problems  common  to  most  Federal 
agency  evaluations.   He  noted,  for  instance,  that  some  evaluation  manage- 
ment systems  are  very  disorganized;  many  RFPs  are  poorly  written,  and 
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frequently  the  academics  who  do  evaluation  research  for  the  Government 
orient  it  more  to  support  their  basic  research  or  training  needs  than 
to  serve  governmental  information  and  analysis  needs.   He  noted  that 
the  most  effective  evaluations  are  sponsored  by  agencies  which  have 
special  program  evaluation  offices,  with  a  technically  competent  staff, 
linked  closely  to  the  agency's  decisionmaking  center. 

There  is  a  special  program  evaluation  office  in  the  Office  of 
Education.   It  has  responsibility  for  evaluation,  planning,  and  budgeting. 
The  Education  office  has  a  staff  of  20  people  and  a  budget  of  $13  million  tc 
evaluate  Federal  education  programs  costing  $7  billion. 

The  following  procedures  are  used  to  do  evaluations  in  the  Office 
of  Education.   The  evaluation  staff  develops  a  list  of  priority  evalua- 
tions which  can  be  funded  from  the  basic  evaluation  research  budget. 
Agency  staff  familiar  with  evaluation  research  and  social  research 
methods   draw  up  a  technically  sophisticated  evaluation  design  before 
issuing  RFPs  to  avert  the  problem  of  proposal  preparation  and  sub- 
mission by  technically  unqualified  performers.   After  proposals  are 
submitted,  an  intra-agency  technical  committee  reviews  proposals, 
selects  the  contractor,  and  closely  monitors  the  research  to  ensure  that 
it  meets  the  agency's  needs.   Monitors  pay  especially  close  attention 
to  the  methodology  used  in  the  evaluation  study,  since  frequently  evalua- 
tions can  be  discredited  for  faulty  methodology. 

After  the  evaluation  study  is  completed,  the  agency  project  monitor 
prepares  a  memo  summarizing  the  evaluation  report  and  identifying  any 
policy  implications  which  might  have  been  generated.   The  memo  is  sent, 
via  the  Commissioner  of  Education,  to  the  Office  of  Management  and  Budget. 
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F.  Concluding  Comments  on  "Evaluation  Evaluation" 

Allen  Schick's  presentation  on  "evaluating  evaluation"  was  a 
succinct  commentary  on  some  of  the  most  important  themes  which  had  been 
raised  in  previous  sessions.   He  noted  that  although  Federal  program 
evaluation  budgets  have  continued  to  increase,  program  evaluation  suffers 
from  deficiencies  of  conceptualization,  methodology,  and  use  in  policy- 
making. 

Dr.  Schick  said  that  agencies  and  the  Congress  are  confronted  with 
numerous  problems  in  doing  and  using  evaluations.   First,  it  may  be 
advisable  for  agencies  to  separate  evaluation  from  budgeting,  since 
agencies  usually  try  to  maximize  their  budgets  and  retain  programs. 
Second,  because  of  its  critical  adversary  nature,  evaluation  might  be 
performed  best  in  a  separate  inspector  general  office.   Third,  agencies 
tend  to  avoid  objective  evaluation  because  they  are  the  captive  of 
their  programs  and  would  like  to  believe  that  programs  are,  indeed, 
benefitting  their  clients.   Fourth,  evaluation  is  very  difficult  to  do. 
Data  collected  in  the  normal  course  of  managing  a  program  are  insufficient. 
It  is  exceedingly  difficult  to  collect  and  interpret  hard-to-measure 
information  on  program  impacts.   Fifth,  administrators  as  well  as  members 
of  Congress  may  sometimes  shy  away  from  using  the  results  of  evaluations 
since  evaluation      often  indicates  that  programs  are  ineffective. 
This  problem  is  compounded  since  most  evaluations  are  done  on  innova- 
tive social  programs,  not  on  long-established  programs  which  may  profit 
more  from  good  evaluation.   Sixth,  at  times  evaluation  of  new  programs 
takes  precedence  over  evaluation  of  past  enactments.   This  is  all  the 
more  significant  because  committees  can  become  more  "program  advocates" 
than  program  overseers.   Therefore  congressional  oversight  may  suffer. 
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Dr.  Schick  remarked  that  the  Congress  is  now  gearing  up  to  perform 
a  better  oversight  role.   However,  he  contended  that  the  Congress  should  not 
as  a  rule  get  directly  into  the  business  of  conducting  evaluations.  Congress 
should  be  concerned  with  legislative  oversight;  this  is  not  analogous  to 
program  oversight,  which  should  be  the  responsibility  primarily  of  executive 
branch  agencies. 

The  Congress  can  assist  in  improving  legislative  oversight  and  the 
use  of  program  evaluation,  Dr.  Schick  said,  by  taking  the  following  steps: 

1.  Improve  the  identification  of  goals  and  meaning  of  program 
objectives  and  congressional  intent  in  legislation  mandating 
new  programs ; 

2.  insist  that  evaluations  be  kept  honest  and  that  impacts,  not 
only  process  issues,  be  measured; 

3.  permit  more  time  for  the  conduct  of  complex  longitudinal 
evaluations  of  ongoing  programs; 

4.  insist  that  evaluations  be  concise  and  readable  to  the  layman; 
and 

5.  make  greater  efforts  to  use  evaluation  information  in  the 
authorization  and  appropriations  process. 
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"FEDERAL  PROGRAM  EVALUATION:   THE  PERSPECTIVE  FROM  OMB,"  A  PAPER 
PREPARED  FOR  THE  AMERICAN  POLITICAL  SCIENCE  ASSOCIATION  BY  CLIFFORD 
W.  GRAVES  AND  STEFAN  A.  HALPER,  USED  AS  A  BASIS  FOR  REMARKS  PRESENTED 
BY  CLIFFORD  W.  GRAVES,  OFFICE  OF  MANAGEMENT  AND  BUDGET 


This  paper  discusses  the  principal  issues  relating  to  governmental 
evaluation  activities  and  traces  the  evolution  of  OMB's  response  to  them 
since  the  1950' s.   It  concludes  with  a  description  of  our  current  approach 
and  some  challenges  to  the  professional  and  academic  evaluation  community. 

The  main  thesis  of  the  paper  is  that  the  definition  and  role  of  eval- 
uation in  which  governmental  activity  is  conducted,  to  label  various  ap- 
proaches, successes  or  failures  whithout  relating  them  to  their  contemporary 
context  is  misleading. 

A  secondary  thesis  is  that  there  has  been  no  revolution  in  governmental 
evaluation;  "evolution"  is  a  more  appropriate  description.   Current  practices 
build  on  the  lessons   of  the  past,  and  future  approaches  will  follow  the 
same  pattern. 

Finally,  the  authors  are  concerned  about  the  ability  of  the  evaluation 
profession  to  respond  to  the  current  challenge.   There  is  a  noticeable 
divergence  between  the  demands  of  the  governmental  decisionmaking  process 
and  the  techniques  and  processes  of  the  evaluator. 


*First  delivered  at  the  Midwest  Political  Science  Association  meeting, 
Chicago,  Illinois,  May  1,  1975.   Remarks  delivered  were  based,  in  part, 
on  this  paper. 
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The  Importance  of  Program  Evaluation 

In  the  current  period  of  budgetary  stringencies,  government  decision- 
makers are  continually  faced  with  perplexing  and  diffcult  options.   The 
dollar  purchases  fewer  goods  and  services  today  than  in  past  years,  and 
resources  for  new  programs  must  be  secured  at  the  expense  of  existing  ones. 
More  than  ever  before,  decisions  concerning  programmatic  priorities  should 
be  based  upon  as  broad  and  objective  an  information  base  as  possible.   Program 
evaluation  is  a  process  that  enhances  objective  decisionmaking;  it  provides 
a  unique  input  to  decisions  bearing  on  effective  program  management,  the 
relative  effectiveness  of  alternative  approaches,  and  the  substantive  im- 
pact of  government  activities. 

Given  the  rising  demands  on  government  and  shrinking  resources  avail- 
able to  it,  the  role  of  evaluation  has  become  critical.   Both  as  a  manage- 
ment tool  and  a  measure  of  impact,  evaluation  may  come  closer  than  any 
other  process  to  providing  the  objective  information  central  to  decision- 
making. 

Working  Definition 

Like  "planning"  and  'coordination,"  "evaluation"  is  a  loosely  used 
term  in  governmental  circles;  it  lacks  any  universally  accepted  definition. 
Like  the  other  two  terms,  evaluation  is  very  difficult  to  oppose  in  prin- 
ciple, even  when  one  is  not  sure  what  it  is.   For  the  purpose  of  OMB 's 
mission  (and  this  is  applicable  to  most  of  the  domestic  Federal  agencies) 
program  evaluation  can  be  defined  as  a  management  process  that  seeks  to 
systematically  analyze  Federal  programs  (or  their  components)  to  determine 
the  extent  to  which  they  have  achieved  (or  are  achieving)  their  objectives. 
The  essential  characteristic  which  distinguishes  evaluation  from  other  forms 
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of  research  is  intended  use:   It  is  decision-oriented. 

Issues 

Three  interrelated  issues  perennially  confront  0MB  and  others  concerned 
with  governmental  evaluation  policy  and  practices: 

—  selection  and  design  of  evaluation  projects; 

—  utilization  of  evaluation  findings; 
"evaluability"  of  governmental  activities. 

They  will  serve  as  reference  points  for  the  discussion  which  follows: 

Projects  Selection  and  Design.   The  planning  of  evaluation  programs — 
the  process  of  determining  what  to  evaluate,  when  to  evaluate  it,  and  how 
to  go  about  conducting  the  evaluation — is  one  of  the  less-developed  areas 
of  public  management.   Yet,  this  fundamental  first  step  in  the  evaluation 
process  can  make  or  break  project  success.   A  1974  0MB  survey  of  current 
agency  practices  revealed  a  wide  variety  of  approaches;  we  suspect  that 
this  variety  has  changed  little  from  the  situation  15  years  ago,  when  the 
first  attempts  at  more  rational  planning  were  undertaken. 

Use  of  Evaluation  Findings.   No  professional  likes  to  see  his  work 
gathering  dust  on  a  bookshelf.   Yet,  this  fate  befalls  many  evaluation 
project  reports  of  both  high  and  low  technical  quality.   At  0MB,  this  is 
disturbing  both  from  a  budgetary  and  program  management  standpoint. 

The  number  of  potential  users  of  evaluation  findings  is  growing. 
The  governmental  decisionmaking  process  is  more  pluralistic  and  open 
than  even  before,  and  the  demand  for  objective  information  has  also 
increased-.   The  issue  of  who  receives  evaluation  findings  and  how  the 
findings  are  used,  therefore,  looms  very  large  among  our  concerns. 
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Evaluability 

Wholey  and  others  argue  that  much  if  not  most  of  what  the  Federal  Govern- 
ment does  is  not  "evaluable;"  i.e.,  the  rigorous  analysis  called  for  in  our 
definition  is  either  impossible  or  yields  no  new  useful  information.  55/Whether 
or  not  one  fully  supports  that  conclusion,  the  fact  remains  that  many  govern- 
mental activities  have  no  basis  in  testable  theory,  lack  measurable  objectives, 
or  exist  beyond  effective  management  control. 

This  is  not  a  new  phenomenon,  indeed,  the  newer  programs  initiated  by 
the  Congress  and  the  Administration  are  much  better  designed  than  older  ones 
were.   However,  the  issue  of  how  to  design  and  manage  a  program  so  that 
evaluation  is  possible  is  far  from  being  resolved. 

Evolution  of  the  OMB  Approach 

In  reviewing  the  executive  branch  and,  more  specifically,  OMB's 
interest  in  program  evaluation,  two  significant  trends  stand  out.   The 
first  of  these  is  the  changing  focus  of  evaluation,  from  efficiency  in  the 
1950's;  to  statistical  program  output  in  the  1960's;  to  a  more  generalized 
program  performance  measure  in  the  1970' s. 

At  the  same  time,  OIlB's  approach  to  evaluation  move  from  one  of  no 
systematic  interest  in  the  1950 's  to  an  internally  focused  system  during 
the  Planning  Program  and  Budgeting  System  (PPBS)  era,  and  currently  to  an 
external  or  agency  focus  in  the  early  1970' s. 


55/  Dr.  Joseph  S.  Wholey,  Staff  member  of  the  Urban  Institute,  Washington, 
D.C. 
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Background  (prior  to  1965) 

During  the  Eisenhower  Administration,  BOB  was  primarily  concerned  with 
operational  efficiency.   This  reflected  the  heavy  emphasis  within  BOB  on 
numbers  rather  than  program  substance  and  policy.   To  a  large  extent,  there 
was  no  such  thing  as  formalized  program  evaluation.   The  budget  examiners 
did  special  analysis,  had  frequent  contact  with  the  agencies,  and  further, 
the  Federal  budget  was  structured  along  organizational  lines,  rather  than 
programmatic  lines. 

In  the  early  1960 's  under  the  Kennedy  Administration,  the  focus  of  the 
budget  examiners  shifted  toward  the  substance  of  programs.   However,  0MB 's 
interest  in  program  evaluation  per  se  remained  unstructured.   With  the 
Great  Society  and  the  creation  of  large  numbers  of  new  programs,  0MB 's 
operational  structure  became  overburdened.   At  the  same  time,  with  the 
growth  of  a  new  professionalism  in  the  public  administration  field,  new 
forms  of  relating  expenditures  to  effects  became  known.   This  trend  was 
largely  responsible  for  the  development  of  the  PPBS  in  1965. 

The  PPBS  ERA  (1965-1970) 

The  Planning  Program  and  Budgeting  System,  first  operationalized  in 
the  Department  of  Defense,  was  primarily  directed  at  improving  the  overall 
management  of  Federal  Government  resources.   It  attempted  to  link  program 
funding  decisions  with  program  actions  and  assessments  of  benefits.   The 
key  elements  of  the  PPBS  were: 

—  a  program  structure  which  organized  agency  activities  in  terms 
of  objectives  or  output  and  costs; 

—  program  memoranda  which  contained  decisions  on  current  major 
issues  and  the  rationale  for  those  decisions,  including  of  options; 
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—  specific  analytic  studies  which  analyzed  the  issues;  and 

—  program  and  financial  plans  which  presented  in  tabular  form  future 
cost  implications  of  major  program  decisions. 

Reduced  to  its  essentials,  the  PPBS  was  an  attempt  to  provide  for 

a  more  orderly  and  reliable  decision  process  by  clarifying  and  making  more 

explicit  the  reasons  for  decisions. 

Most  important,  PPBS  represented  the  concept  that  government  carries  a 
responsibility  for  achieving  specific  objectives  rather  than  simply  trying  to 
make  general  improvements  in  a  particular  area.   This  change  of  orientation 
implies  a  close  examination  of  program  impact  as  a  part  of  the  public  management 
process. 

From  the  perspective  of  program  evaluation,  PPBS  represented  the  first 
significant  systematic  approach  to  the  development  and  use  of  program  evalu- 
ation as  a  part  of  the  public  decisionmaking  process.   Program  evaluations 
were  directed  toward  issues  which  were  cross-cutting  in  nature;  the  evaluations 
were  designed  in  the  form  of  specific  analytic  studies.   Program  memoranda 
were  then  developed  which  used  evaluation  information  in  policy  analysis  and 
decision  options.   The  primary  users  of  PPBS  evaluation  information  were  OMB 
and  Administration  policymakers,  rather  than  agency  program  managers  and  the 
Congress. 

The  system  was  not  specifically  decision-driven  except  to  the  extent  that 
evaluative  studies  were  undertaken  in  priority  issue  areas.   There  was  no 
established  process  for  structuring  programs  in  a  manner  which  would  neces- 
sarily render  them  evaluable. 


71-867  O  -  76 
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Observations  About  the  PPBS  ERA 

Many  people  have  speculated  on  the  reasons  for  the  demise  of  PPBS.   The 
precise  reasons  for  its  decline  are  not  relevant  to  us  here.   However,  what 
is  relevant  is  the  attempt  to  deal  with  the  Federal  management  process  and 
the  integral  nature  of  evaluative  activity  within  that  process.   Many  of  the 
underlying  premises  of  PPBS  are  taken  as  givens  today;  for  example,  the  focus 
on  program  and  program  impacts.   PPBS  also  contributed  much  to  the  development 
of  professional  skills  within  government  to  evaluate  and  judge  program  per- 
formance.  Program  evaluation  and  measurement  techniques,  even  today,  are 
not  perfect;  but  it  should  be  recognized  that  PPBS  was  the  first  Government- 
wide  management  system  to  focus  on  improving  evaluation  procedures  and  in- 
tegrating evaluation  information  into  the  decision  process.   We  are  still 
learning. 

The  Performance  Management  System  (1970-1972) 

As  the  limitations  of  PPBS  became  clear,  BOB  initiated  a  study  to 
investigate  possible  new  approaches.  The  focus  was  the  possible  inte- 
gration of  the  PPB  and  the  then-existing  budgetary  system. 

The  study  concluded  that  integration  was  possible.   A  system  was 
devised  which  measured  program  effectiveness  in  terms  of  impact  vs.  bud- 
geted resources  consumed.   It  also  provided  for  the  timely  reporting  of 
important  information  to  decisionmakers.   Thus,  the  basic  structure  of  the 
Performance  Management  System  was  developed.   In  late  1970,  the  President 
authorized  0MB  to  proceed  to  test  the  system  by  implementing  it  in  selected 
high  priority  domestic  programs. 
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The  PMS  concept  was  based  upon  the  principles  of  management  by  ob- 
jectives and  accountability  for  results.   It  recognized  that  the  system 
had  to  be  flexible  enough  to  adapt  to  the  variety  of  program  and  organi- 
zational characteristics  found  in  the  Federal  Government.   The  intent  was 
to  develop  and  implement  a  system  based  on  a  broad  concept  without  prescribed 
rigid  formats  for  its  operation.   Essentially,  the  system  concepts  were: 

—  clearly  defined  program  objectives; 

—  a  designated  manager  who  could  be  held  responsible  and  accountable; 

—  identification  of  performance  measures  with  time-phased  targets 
for  performance; 

—  periodic  reporting  of  performance  based  on  the  performance  mea- 
sures and  targets;  and 

—  identification  of  significant  variations  in  performance  to  allow 
for  timely  corrective  action. 

The  PMS  concept  placed  responsibility  on  the  program  manager  for  se- 
lecting and  designing  evaluation  studies.   Information  was  to  be  integrated 
by  him  into  decisions  taken  in  reaching  his  objectives.   Both  process  and 
impact  evaluation  were  contemplated,  although  the  concept  of  evaluability 
did  not  play  a  specific  part  in  program  design. 

The  Performance  Management  System  never  moved  beyond  the  pilot  test 
phase.  With  changes  in  OMB 's  top  management,  PMS  was  reconstituted  with 
a  narrower  focus  in  the  form  of  the  Management  by  Objectives  process  (MBO) . 

Management  by  Objectives  can  be  characterized  as  a  limited  management 
system  for  Presidential  and  Cabinet-level  concerns.   The  MBO  process,  as 
it  was  actually  instituted,  attempted  to  focus  attention  on  a  limited 
number  of  specific  Presidential  objectives.   Agency  heads  were  encouraged 
to  establish  their  own  MBO  systems  for  internal  management. 


266 


However,  in  its  original  conception,  MBO  represented  what  could  be 
characterized  as  another  expression  of  essentially  the  same  interests  and 
objectives  as  those  which  gave  rise  to  PPBS:   the  public  management  process 
used  by  the  Federal  Government.   While  it  was  less  ambitious  and  less  com- 
prehensive than  PPBS,  it  was  more  pragmatic  in  that  it  did  not  attempt  to 
deal  with  all  of  the  activities  of  the  Federal  Government  at  once,  only 
those  that  were  of  particular  interest  to  the  President  and  other  top  policy 
officers. 

The  MBO  process  provides  for  the  establishment  of  objectives  and 
the  periodic  assessment  of  performance  against  those  objectives.   It  did 
not  attempt  to  relate  those  objectives  specifically  to  the  budget  process, 
nor  to  any  programmatic  or  organizational  structure.   It  instead  focused  on 
the  achievement  of  the  objectives  by  whatever  programmatic,  organizational 
other  means  were  available  and  appropriate  in  each  case. 

The  periodic  assessment  of  performance  of  the  objectives  does  not 
represent,  to  a  limited  extent,  an  evaluation  component.   However,  there 
was  an  apparent  conscious  decision  not  to  attempt  to  make  evaluation  of 
performance  an  integral  part  of  the  OMB  role  in  the  MBO  process,  except 
to  monitor  that  performance. 

For  the  most  part,  evaluations  have  been  initiated  by  agency  pro- 
gram managers  or  OMB  Budget  Examiners  to  obtain  a  closer  look  at  programs 
which  may  form  a  part  of  an  MBO  objective.   Evaluations  have  been  systemati- 
cally utilized  to  the  extent  that  they  relate  to  better  program  management. 
Like  the  PPBS  and  PMS,  there  has  been  no  established  procedure  for  ensuring 
program  evaluability .   The  MBO  concept,  as  it  was  actually  implemented,  fell 
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short  of  its  goals  in  much  the  same  way  as  PPBS  but  for  different  reasons. 
The  MBO  process  failed  to  establish  the  management  system  necessary  to  sys- 
tematically develop  and  evaluate  performance  on  the  objectives. 

Refocusing  Evaluation 

The  reassessment  of  OMB 's  interest  in  program  evaluation  concluded  that 

future  OMB  activities  should: 

focus  on  strengthening  the  agencies  capabilities,  rather  than  those 
of  OMB  itself,  and 

—  attempt  to  improve  the  quality  and  use  of  evaluation  information  in 
the  decisionmaking  processes  without  requiring  it. 

In  short,  the  approach  should  be  to  encourage  the  agencies  to  improve 
evaluation  activities  but  not  to  mandate  action  nor  dictate  what  improvements 
should  be  made.   Some  would  call  this  consciousness  raising. 

This  conclusion — that  OMB's  focus  should  be  on  improving  evaluation 
within  the  agencies  rather  than  within  OMB  itself — is  based  on  the  premise 
that  improvement  in  the  quality  and  management  of  evaluation  in  the  agencies 
will  contribute  better  information  to  the  decisionmaking  process  within  the 
agencies,  as  well  as  within  OMB.   It  recognizes  that  primary  responsibility 
for  program  management,  of  which  evaluation  is  a  part,  rests  with  the  agency 
head. 

Current  Thinking 

When  the  evaluation  policy  function  in  OMB  was  reconstituted  in  early 
1974,  its  objectives  and  operating  style  were  based  on  both  the  lessons  of 
the  past  and  the  changing  governmental  decisionmaking  environment.  Writing 
in  Public  Administration  Review,  Lewis  and  Zarb  set  forth  the  definition  of 


evaluation  used  in  this  paper,  and  noted  that: 

.  .  .OMB  takes  the  position  that  each  agency  head  is 
primarily  responsible  for  managing  the  programs  under 
his  jurisdiction  and  that  program  evaluation  consti- 
tutes an  essential  element  of  effective  management.  .  . 
each  Federal  agency  should  be  responsible  for  carrying 
out  its  own  evaluation  program. 

However,  concern  has  been  expressed  as  to  whether  the 
results  obtained  from  many  of  these  efforts  have  been 
as  useful  and  relevant  as  they  might  be  to  major  policy 
decisions  confronted  by  program  managers,  agency  heads, 
the  President  and  Congress. 

Therefore,  OMB  will  initiate  a  multi-agency  cooperative 
effort  to  upgrade  and  strengthen  the  overall  quality  and 
relevance  of  evaluation  carried  out  by  the  various  Federal 
agencies  and  to  assure  that  evaluation  results  are  utilized 
in  arriving  at  major  policy  decisions.  56  / 

To  implement  this  approach,  OMB  now  works  with  the  departments  and 
agencies  collectively  and  individually,  to  improve  their  evaluation  programs 
without  imposing  a  singular  management  or  evaluation  system;  and  with  the 
legislative  branch  on  program  design,  information  sharing,  and  similar  eval- 
uation-related issues  of  common  concern. 

Out  of  this  collegial  approach  to  evaluation  policy  have  come  a  number 
of  concepts  that  relate  to  the  issues  discussed  earlier  in  this  paper: 

—  a  decision-driven  model  for  evaluation  planning;  and 

—  an  evaluation  component  for  new  Federal  programs. 


Sft/  Lewis,  Frank  L.  and  Frank  G.  Zarb,  (Office  of  Management  and  Budget), 
Federal  program  evaluation  from  the  OMB  Perspective.  Public  Adminis- 
tration Review,  v.  34,  no.  4,  July /August  1974:   312. 


269 


Decision-Driven  Model 

A  program  evaluation  project  can  be  considered  useful  if  it  is  timely, 
accurate,  and  relevant,  i.e.,  it  is  available  to  a  decisionmaker  at  a  time 
when  a  decision  is  to  be  made,  it  is  methodologically  sound,  and  contains 
reliable  information  relevant  to  the  decisionmakers'  needs. 

Based  on  information  from  a  recent  OMB  survey  of  Federal  evaluation 
activities,  we  identified  three  prevailing  approaches  used  by  most  agencies 
to  plan  evaluation  projects.   Each  of  the  commonly  used  approaches,  i.e., 
(1)  projects  undertaken  on  direct  demand,  (2)  projects  undertaken  on  the 
basis  of  a  cyclical  review  of  programs,  and  (3)  projects  undertaken  as  a 
result  of  attempts  to  conceptualize  information  needs,  have  one  common 
characteristic — they  rarely  involve  explicit  consideration  of  the  decisions 
they  should  be  designed  to  support  or  identification  of  involved  decision- 
makers.  Consequently,  utilization  of  the  studies  in  arriving  at  program  de- 
cisions is  often  fortuitous. 

Evaluation  planners  must  focus  on  major  decision  alternatives  or 
subject  areas  around  which  decisions  are  anticipated  when  developing  pro- 
posed evaluation  projects.   Further,  decisionmakers,  program  managers,  and 
other  staff  units  must  review  these  proposed  projects  and  approve  or 
disapprove  the  evaluation  plan  based  on  the  sufficiency  and  timeliness  of 
information  to  be  available  for  their  individual  decision  needs.   This 
approach,  which  we  have  called  the  decision-driven  model,  has  several  ad- 
vantages: 

—  it  provides  greater  clarity  in  the  objectives  of  individual 
projects;  and 

—  decisionmakers  are  better  prepared  to  receive  and  use  evaluation 
findings  for  specific  decisions. 
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Operationally,  the  model  is  difficult  because  of  the  discipline  re- 
quired of  both  decisionmakers  and  evaluators,  and  because  of  the  multiple 
decisionmaker  mode  that  characterizes  the  Federal  Government.   However, 
these  problems  do  not  negate  the  value  of  the  approach. 

This  model  not  only  guides  project  selection  and  timing,  it  also 
should  influence  project  design.   The  central  criticism  with  this  approach 
is  that  decisionmakers,  even  in  the  best  of  circumstances,  rarely  project 
their  evaluation  information  needs  beyond  a  6  or  9  month  time  frame;  and  most 
extant  evaluation  techniques  cannot  yield  reliable  information  in  so  short  a 
time.   Therefore,  with  the  exception  of  those  cyclical  decision  processes 
like  the  budget,  where  decisions  can  be  anticipated  a  year  or  more  in 
advance,  serious  questions  arise  as  to  the  usefulness  of  the  studies. 

Since  most  decisions  will  be  made  without  adequate  evaluation  infor- 
mation, and  since  the  public  and  the  Congress  demand  accountability  for 
government  spending  and  decisionmaking,  it  is  critical  that  the  academic 
and  research  communities  develop  reliable  and  accurate  rapid  turnaround 
techniques.   The  limited  progress  which  has  been  made  in  this  area  falls 
into  two  categories.   Preassessment  techniques  tell  decisionmakers  whether 
their  information  needs  can  be  met  through  evaluation;  this  procedure 
offers  indirect  performance  data.   A  second  approach,  based  on  the  military 
quick  reaction  capability  utilizes  quick  survey  procedures  and  panels  of 
experts  to  provide  gross  policy  direction — e.g.,  terminate/reauthorize, 
increase/decrease  program  resources.   We  are  suggesting  that  there  is  a 
pressing  need  for  additional  methodological  research  to  address  this  problem. 
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Creative  talent  is  needed  in  this  area,  for  breakthroughs  here  can  affect 
the  entire  spectrum  of  government  decisionmaking. 

Evaluation  Components  in  New  Programs 

The  development  of  adequate  rapid  feedback  procedures  represents,  in 
effect,  a  tactical  step  in  providing  decisionmakers  with  generally  reliable 
interim  information.   On  a  broader  scale,  it  is  OMB's  objective  to  include 
appropriate  evaluation  components  into  programs  at  their  inception  through 
both  legislative  and  administrative  means.   This  will  make  evaluation  easier 
and  more  useful  later. 

On  the  legislative  side,  working  relationships  have  been  developed  with 
legislators  and  the  key  committee  staffs  in  both  Houses  to  gain  agreement  on 
the  nature,  structure  and  actual  legislative  language  to  be  used  in  bills 
so  that  information  gathering,  as  part  of  the  evaluation  process,  is  ini- 
tiated as  soon  as  programs  begin.   An  OMB  staff  paper  now  being  reviewed 
proposes  that  six  elements  be  taken  into  consideration  when  drafting  new 
legislation. 

1.  State  program  objectives  in  a  measure  form. 

2.  Seek  agreement  between  legislators  and  administrators  on  the 
criteria  that  will  be  used  (a)  to  measure  the  degree  to  which 
programs  reach  their  objective  and  (b)  to  measure  the  degree 
to  which  programs  impact  the  problem. 

3.  Ensure  that  extant  methodology  is  capable  of  generating  the 
information  program  managers  required  for  decision. 

4.  Ensure  adequate  staff  and  funds  are  available  to  perform  the 
evaluation. 

5.  Ensure  that  there  are  clear  provisions  for  the  flow  of  evaluation 
information  from  evaluation  to  decisionmaker. 

6.  Establish  who  will  use  the  evaluation  information. 
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On  the  administration  side,  the  paper  proposes  that  OMB  instruct 
the  agencies  to  include  program  objectives  statements  and  evaluation  plans 
for  new  programs  or  proposed  legislative  initiatives. 

The  review  of  proposed  legislation  in  OMB  is  currently  channeled 
through  the  Legislative  Reference  Division  to  the  appropriate  program 
divisions.   There,  a  budget  examiner  or  management  associate  review  pro- 
posals for  compliance  with  certain  statutory  and  Executive  Order  require- 
ments.  Civil  Rights,  environmental  impact,  and  cost  analysis  are  three 
such  requirements.   The  OMB  circular  would  add  the  evaluation  plan  to  this 
list. 

Much  of  the  impetus  for  the  proposal  to  build  evaluation  into  new 
programs  and  other  current  policy  initiatives  stems  from  a  survey  of 
Federal  evaluation  activities  performed  by  OMB  in  the  spring  of  1974.   The 
survey  was,  in  effect,  a  first  step  in  understanding  and  strengthening 
evaluation  activity  in  the  major  domestic  agencies.   It  has  provided  baseline 
data  on  the  nature  and  scope  of  current  agency  practices.   The  survey: 

—  identified  organizational  settings  and  priorities  of  units 
for  which  program  evaluation  represents  a  major  and  continuing 
responsibility; 

—  discussed  the  nature  of  operating  relationships  between  evaluation 
units  and  units  involved  in  related  information  producing  activities. 

—  described  the  processes  employed  to  plan,  coordinate  and  disseminate 
evaluation  studies;  and 

—  obtained  expenditure  and  resource  information. 

An  update  is  now  in  progress;  this  time  the  survey  is  being  jointly 
conducted  by  OMB  and  the  General  Accounting  Office.   With  the  help  of  a 
panel  of  senior  evaluation  officials  (generally  the  Assistant  Secretary 
level)  we  have  sought  to  relate  the  survey  finding  to  the  management  issues 
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surrounding  the  decision-driven  model  described  earlier. 

We  have  augmented  this  process  through  symposia  and  conferences  where 
we  have  had  a  valuable  exchange  of  ideas  with  the  academic  and  research  com- 
munities on  the  use  of  evaluation  in  public  policy  formulation. 

Future  analytic  efforts  are  planned  on  evaluation  and  the  budget  process 
and  a  review  of  evaluation  methodology  in  selected  subject  areas.   The  papers 
are  intended  as  a  mechanism  for  eliciting  throughtful  comment  on  the  issues 
and  as  reference  points  for  OMB  initiatives.   For  example,  specific  action  steps 
identified  from  the  current  papers  are: 

—  An  OMB  Circular  to  provide  general  guidelines  for  managing 
evaluation  activities  and  building  evaluation  components  into 
new  programs  (now  drafting). 

—  In  depth  assessments  of  selected  agency  evaluation  activities 
(to  begin  in  July  1975). 

—  A  comprehensive  analysis  of  the  evaluation  training  needs  of 
Federal  executives,  program  managers,  and  evaluation  staffs 
(to  begin  June  1975). 

—  On  a  very  limited  basis,  some  technical  assistance  has  been  pro- 
vided to  agencies  (on  request)  in  structuring  evaluation  systems. 

—  We  are  providing  some  technical  assistance  to  the  OMB  program 
divisions  on  evaluation  projects  and  legislative  issues,  on  re- 
quest.  Labor's  evaluation  of  the  CETA  program  is  an  example. 

Conclusion  A 

The  current  OMB  effort  has  been  structured  so  that,  over  time,  it 
should  result  in  a  significant  improvement  in  the  contribution  of  evaluation 
to  the  policymaking  process. 

While  our  primary  responsibility  is  to  improve  the  quality  of  evaluation 
information  used  by  the  President  and  by  OMB,  the  approach  I  have  described 
will  benefit  the  Congress  and  the  growing  numbers  of  decisionmakers  in 
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State  and  local  governments.   The  response  to  the  approach  so  far  has 
been  encouraging. 

The  professional  evaluation  community  will  find  their  work  subjected 
to  more  scrutiny  from  Federal  sponsors  than  in  the  past.   Agency  managers 
are  becoming  more  sophisticated  in  the  use  of  evaluation,  Congress,  GAO, 
OMB,  and  others  will  be  increasing  their  demand  for  timely  and  reliable 
evaluation  information. 

As  noted  earlier,  some  better  techniques  are  needed  to  satisfy  this 
demand.   More  important,  the  professional  community  will  have  to  make  sure 
their  products  can  stand  the-  test  of  implementation. 
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DISCUSSION 

In  response  to  questions  raised  during  the  discussion,  Mr.  Graves 
reiterated  that  OMB  and  agency  budget  officers  do  not  always  use  eval- 
uation information  in  planning  and  programming.   This  happens  because 
information  is  not  always  timely  or  adequate. 

Several  clarifications  were  made  about  separating  the  evaluation 
functions  of  setting  priorities,  conducting  evaluations,  and  using 
evaluation  information.   It  was  noted,  for  instance,  that  there  is  a 
separate  office  of  planning  in  the  Department  of  Health,  Education, 
and  Welfare,  which  recommends  priorities  for  evaluation  studies  to 
be  conducted  by  different  offices.   However,  recommendations  for 
evaluations  also  filter  up  from  the  line  agencies  to  the  planning  of- 
fice. 

One  problem  is  common  to  all  executive  agency  evaluations.   This 
is  the  absence  of  a  central  reporting  mechanism  to  collect  and  dis- 
tribute evaluation  reports.   According  to  Mr.  Graves,  agencies  should 
devote  more  attention  to  developing  evaluation  libraries  or  informa- 
tion dissemination  mechanisms. 

Mr.  Graves  also  remarked  that  it  would  be  advantageous  to  en- 
courage Federal  agencies  to  make  all  evaluation  studies  public.   This 
would  enable  critics  to  double  check  the  work  of  evaluation  researchers 
with  respect  to  assumptions  and  methodology  used,  data  gathered,  and 
conclusions  reached. 

A  member  of  the  audience  raised  a  very  important  point  tangential- 
ly  related  to  the  seminar  theme.   Do  the  OMB  and  Federal  agencies  use 
policy  analysis?  Mr.  Graves  replied  that  with  the  exception  of  the 
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Defense  Department  executive  agencies  do  not  make  much  use  of  policy 
analysis  which  incorporates  the  findings  of  program  evaluations.   Sev- 
eral social  policy  agencies  have  attempted  to  use  policy  analysis  for 
planning  and  budgeting  purposes,  but  these  attempts  have  been  futile. 
The  final  issue  raised  after  the  OMB  presentation  related  to  how 
the  OMB  might  develop  statements  or  objectives  of  program  goals  or 
objectives  for  use  in  evaluation.   According  to  Mr.  Graves  agencies 
and  the  OMB  use  four  information  sources: 

-  legislative  language, 

-  legislative  history, 

-  consultations  with  committee  staff,  and 

-  discussions  with  White  House  and  agency  officials. 

In  conclusion,  Mr.  Graves  remarked  that  OMB  has  mixed  reactions 
to  legislatively  mandated  statutory  requirements  for  program  evalua- 
tions.  Some,  he  said,  are  too  stringent  and  impossible  to  fulfill. 
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'CURRENT  EVALUATION  ACTIVITIES  IN  THE  GENERAL  ACCOUNTING  OFFICE,' 
KEITH  E.  MARVIN,  GENERAL  ACCOUNTING  OFFICE 


My  presentation  will  cover  program  evaluation  activities  which  in- 
clude evaluation  methods,  evaluation  synthesis,  and  evaluation  and  mod- 
eling sources.   First,  with  regard  to  methods,  we  are  developing  stan- 
dards and  guidelines  for  program  evaluation  and  doing  other  work  closely 
related  to  what  Harrison  Fox  will  be  talking  about  later.   Our  synthesis 
activity  is  where  we  will  be  putting  together  and  refining  evaluation 
information  related  to  specific  program  areas  or  issues.   Incidentally, 
I  don't  want  to  leave  the  impression  that  all  of  the  GAO  work  on  pro- 
gram evaluation  is  done  in  this  staff.   All  GAO  divisions  are  involved 
in  program  evaluation  and  the  handout  I  have  provided  to  you  lists  about 
60  reports  issued  in  1973  and  1974  which  contain  or  deal  with  evaluation 
of  Federal  programs.   (See  Appendix  ,J      One  of  the  handouts  describes 
9  of  these  reports  in  more  detail  showing  the  kind  of  constructive  crit- 
icisms which  GAO  has  made  to  make  program  evaluation  more  valuable  in 
decisionmaking  in  a  variety  of  Federal  programs.   (See  Appendix*) 

Basic  Evaluation  Information 

Our  major  activity  since  January  1975  has  been  to  develop  and  con- 
duct jointly  with  the  Office  of  Management  and  Budget  an  inquiry  into 
department  and  agency  evaluation  activities.   As  chart  1  indicates,  we 
are  collecting  information  on  recently  completed  evaluations,  on  in- 
process  evaluations  and  on  the  plans  at  least  by  program  area  for  eval- 
uation to  be  conducted  in  the  next  fiscal  year.   As  the  chart  also  in- 
dicates, we  also  cover  the  evaluation  process  in  the  inquiry  and  we  in 
GAO  will  be  interested  in  whether  those  processes  are  working  properly. 
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However,  if  the  program  described  by  Mr.  Graves  works  out  well  perhaps 
we  will  not  need  to  spend  as  much  time  worrying  about  whether  the  pro- 
cess is  working  as  in  providing  better  information  to  the  Congress. 

The  kind  of  information  that  we  will  be  able  to  provide  to  com- 
mittees is  illustrated  by  the  loose  leaf  book  which  I  have  here  today 
covering  the  evaluation  information  obtained  on  HEW  and  Department  of 
Labor  programs.   The  booklet  is  organized  into  sections  for  the  major 
areas  such  as  education  and  health. 

Examples  of  Evaluation  Information 

Chart  2  is  an  illustration  of  the  kind  of  information  entered  in 
the  booklet  which  I  just  described.   On  the  right  is  the  title  of  an 
evaluation  report  and  the  date  of  issue.   This  is  associated  with  the 
title  of  the  program  evaluated  in  the  center  column  and  on  the  left  is 
shown  the  legislation  which  authorizes  the  program  evaluated.   We  have 
found  in  some  preliminary  discussions  with  committee  staffs  that  it  is 
helpful  to  them  to  be  able  to  select  the  information  pertaining  to  par- 
ticular legislation.   We  have  also  found  that  while  the  titles  of  the 
evaluations  give  them  some  indication  of  the  things  that  they  might  want 
to  look  deeper  into,  they  need  more  information  in  order  to  make  these 
selections. 

Chart  on  Descriptive  Abstracts  of  an  Evaluation 

Chart  3  illustrates  for  one  of  the  evaluations  in  the  Department 
of  Labor  list  how  the  evaluation  can  be  abstracted  to  show  such  things 
as  the  scope.   Other  information  which  can  be  included  is  the  location 
where  the  study  can  be  obtained,  the  period  of  time  covered  by  the  eval- 
uation, and  possibly  any  major  conclusion  reached. 
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We  will  consider  also  whether  certain  features  of  the  methodology 
need  to  be  included  in  this  kind  of  information.   For  example,  users 
of  the  evaluation  may  be  particularly  interested  in  the  goals  speci- 
fied for  the  program  which  normally  are  the  basis  for  the  choice  of 
effectiveness  measures. 

Chart  on  Criteria  for  Measures 

In  our  work  on  standards  for  evaluation,  we  have  looked  into  pos- 
sible criteria  for  the  choice  of  measures.   Chart  A  shows  a  number  of 
such  criteria  which  are  not  original  with  us,  but  have  been  specified 
by  a  number  of  evaluation  staffs  working  in  the  field.   When  an  eval- 
uation is  being  planned,  there  are  trade-offs  which  must  be  made  among 
these  criteria;  e.g.,  the  simplest,  lowest  in  cost  measures  may  not  be 
the  most  sensitive  measures  or  the  ones  which  measure  unique  features 
of  a  program  best.   The  evaluation  staff  will  need  to  understand  the 
users  need  to  make  these  trade-offs  efficiently.   This  list  is  an  ex- 
ample also  of  the  kind  of  standards  and  criteria  which  our  Evaluation 
and  Modeling  Sources  staff  will  be  using  in  appraising  the  quality  of 
selected  evaluations  and  models  which  are  of  particular  interest  to 
committees.  We  have  found  in  our  preliminary  discussions  with  commit- 
tee staffs  that  when  they  select  from  the  inventory,  evaluations  of 
interest,  that  they  will  be  interested  in  our  help  to  determine  whether 
these  evaluations  can  be  trusted. 

Evaluation  and  Modeling  Sources  System 

Most  of  the  staff  on  these  activities  are  systems  analysis  types 
and  we  have  viewed  this  activity  as  a  system.   As  Chart  5  shows,  we 
begin  with  the  evaluations  or  models,  where  they  are  in  the  agencies, 
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EVALUATION  INFORMATION  SERVICE  SYSTEM 
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and  have  established  a  two-way  communication  between  our  staff  and  all 
of  these  major  staffs.   We  also  have  a  continuous  communication  with 
our  resident  audit  staffs  at  the  agency  sites.   This  establishes  a  ba- 
sis for  us  to  monitor  on  a  continuous  basis  any  evaluation  or  model  or 
any  evaluation  process  if  there  is  a  need  to  do  so.   This  is  also  the 
relationship  through  which  we  have  conducted  the  recent  joint  Office 
of  Management  and  Budget  and  GAO  inquiry  into  agency  evaluation  activ- 
ities, which  has  provided  a  basic  file  of  program  evaluation  informa- 
tion. 

As  I- have  already  indicated,  we  are  able  from  this  basic  file  of 
information,  to  provide  to  any  interested  committee  a  list  of  the  eval- 
uations which  have  been  conducted  of  programs  authorized  under  par- 
ticular legislation.   When  we  go  beyond  that  to  assist  the  committee 
in  appraising  the  quality  of  any  evaluation  or  mode,  we  will  also  call 
on  outside  experts  to  supplement  the  knowledge  or  skills  of  our  in- 
house  staff.   As  we  see  it,  this  process  will  build  an  inventory  of  the 
most  relevant  and  highest  quality  evaluation  information  which  can  be 
structured  based  on  the  needs  of  the  various  congressional  users,  to 
provide  tailored  quick  response  evaluation  products  for  these  users. 

In  addition  to  the  congressional  users,  GAO  audit  groups  will  use 
this  information  in  reviews  of  program  results.   We  expect  that  exec- 
utive agency  evaluation  staffs  may  wish  to  use  this  inventory  to  look 
for  evaluations  of  programs  which  may  be  closely  related  to  the  ones 
they  are  evaluating.   There  are  a  number  of  State  evaluation  units  in 
existence  and  we  expect  that  they  may  find  this  inventory  useful  as 
well.  We  expect  that  will  be  a  two-way  street  and  eventually  expect 
to  have  in  our  own  inventory  the  best  State  level  evaluations  which 
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are  available,  particularly  those  which  are  evaluations  of  Federal 
grant  programs. 

We  believe  the  evaluation  and  modeling  inventory  I  have  described 
will  have  both  direct  use  to  the  Congress,  and  will  have  indirect  bene- 
fits in  better  analysis  and  evaluation  for  the  Congress. 
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DISCUSSION 

The  discussion  following  the  GAO  presentation  centered  on  the  expected 
utility  of  the  joint  GAO/OMB  program  information  system.  Mr.  Marvin  said 
that  there  was  considerable  and  enthusiastic  optimism  regarding  the  genera- 
tion of  program  related  information  required  for  legislative  evaluation 
and  budgeting  purposes.   However,  it  will  be  quite  a  while  before  the 
two  agencies  develop  a  system  which  is  both  feasible  and  tailored  to 
legislative  information  needs. 
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'PRACTICAL  APPLICATION  OF  EVALUATION  IN  LEGISLATIVE  PROCESSES",  WILLIAM 
H.  FOSKETT,  CONSULTANT  * 


.  The  following  remarks  on  Program  Evaluation:  A  Manual  for  Legislators 
and  Legislative  Staff  will  discuss: 

°  what  we  mean  by  the  term  "evaluation", 
°   the  concerns  and  perspective  behind  the  manual, 
°  basic  ingredients  involved  in  using  evaluation  for  over- 
sight, and 
°   some  cautions  for  using  evaluation. 
My  remarks  represent  one  co-author's  perspective  which  may  or 
may  not  be  shared  by  the  other  co-author  (Harrison  Fox)  on  any  particular 
point.   These  remarks  will  not  substitute  for  the  manual  itself,  which 
goes  into  more  detail  on  specific  topics  such  as  legislative  language, 
leverage  points,  and  the  need  for  coordinating  evaluation  techniques  with 
legislative  processes.   (The  manual  appears  in  the  Appendix.) 

What  "Evaluation"  Means  Here 


A  variety  of  management  research  and  social  science  research  techni- 
ques come  under  the  rubric  of  "evaluation".   Though  there  are  many  difference 
among  techniques, they  are  alike  in  that  they  are  all  empirical  and  systema- 
tic.  Use  of  these  techniques  is  not  restricted  to  oversight  of  existing 
"programs".   They  may  be  applied  before  a  program  even  exists  (e.g., 
needs  assessment)  or  they  may  cut  across  programs  (e.g.,  policy  analysis). 
Evaluation  techniques  may  yield  evidence  on  program  implementation   (e.g., 
process  evaluation) ,  on  the  relative  effectiveness  of  different  approaches 
to  a  particular  need  or  goal  (e.g.,  planned  variation):  or,  on  the  signi- 
ficance of  program  effects  (e.g.,  impact  evaluation). 

*  Dr.  Foskett  assisted  Dr.  Harrison  Fox,  of  Sen.  Brock's  staff  in  preparing 
the  draft  manual  on  program  evaluation.   Dr.  Foskett  is  chief,  Design  and 
Assessment  Division,  Child  Development  Associate  Consortium,  Washington,  D.C. 
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The  research  tools  involved  In  evaluation  are  the  same  as  those 
used  in  academic  theory  building.   However,  in  evaluation  these  tools  are 
driven  by  the  framework  of  political  issues  and  public  policy  decisions. 
Evaluation  is  a  loose  collection  of  empirical  tools  for  systematically 
gathering  facts,  examining  them  and  using  them  to  think  through  public 
action. 

Concerns  and  Perspective  Behind  the  Manual 

Evaluation  offers  great  potential  for  strenghtening  congressional 
oversight.   But  to  realize  this  potential,  legislators  and  legislative 
staff  need  to  know  how  to  use  the  tools  of  evaluation  in  service  of  legis- 
lative needs.   An  ability  to  apply  evaluation  in  legislative  processes  need 
not  require  technical  training  in  evaluation.   But  an  aspiring  legislative 
user  of  evaluation  must  know  certain  basic  leverage  points  about  the 
substance,  focus  and  quality  of  evaluation  research  in  order  to  integrate 
it  into  legislative  strategies  and  tactics.   The  intention  of  this  legislative 
users  manual  of  evaluation  is  to  acquaint  congressional  readers  with  the 
leverage  points  of  evaluation  in  such  a  way  that  evaluation  can  be  applied 
more  powerfully  and  constructively  in  legislative  work. 

Securing  the  Congressional  Stake  in  Evaluation 

This  evaluation  user's  manual  is  written  on  behalf  of  the  congressional 
stake  in  evaluation.   The  style,  timing  and  substance  of  evaluation  has 
been  influenced  mainly  by  Executive  branch  management  and  policy  planning 
needs  or  by  academic  notions  of  research.   The  evaluation  profession, 
evaluation  research  institutions,  and  the  concepts  of  evaluation  have 
developed  in  service  of  executive  clients,  not  legislative  clients.   Until 
legislative  evaluation  users'  needs  are  strongly  defined  and  established  in 
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practice,  staff  should  give  close  and  thoughtful  attention  to  evalua- 
tion research  done  for  use  in  the  legislative  arena.   The  evaluation 
community  does  not  already  know  how  best  to  serve  legislative  clients. 
It  will  have  to  learn;  users  will  have  to  teach  it. 

The  manual  does  not  present  or  discuss  the  evaluation  activities  of 
the  Executive  Branch.   This  omission  should  not  be  interpreted  as  diminishing 
the  importance  or  necessity  of  evaluation  conducted  on  behalf  of  Executive 
planning  and  management  needs.   Nor  should  it  be  construed  as  minimizing 
the  importance  of  Congressional  and  Executive  cooperation  and  coordination 
in  a  wide  range  of  evaluation  activities.   The  evaluation  activities  of  the 
Executive  Branch  should  not  be  displaced  or  wholly  duplicated  in  the  course 
of  obtaining  Congressionally-oriented  evaluation  services.   But  as  an 
evaluation  user,  Congress  has  some  interests  which  it  can  better  define 
for  itself.   This  manual  seeks  to  open  the  way  for  Congressional  definition 
and  pursuit  of  its  own  distinctive  interests  in  evaluation. 

The  interests  and  perspectives  of  the  Congress  and  the  Executive  branch 
differ  by  design.   The  perspective  of  this  manual  assumes  that  a  balance 
of  power  among  co-equal  branches  of  the  government  is  desirable.   But  a 
variety  of  factors  has  created  an  imbalance  among  the  legislative  and 
executive  branches.   One  factor,  the  information  explosion,  has  flooded 
Congress  with  torrents  of  data.  Another  factor,  the  increasing  complication 
of  our  Nation  as  it  has  grown  in  size  and  experiences,  imposes  many  more 
demands  on  oversight  nowadays. 

The  Executive  Branch  has  been  relatively  better  staffed  and  organized 
to  deal  with  the  complexity  and  the  explosion  of  information     that 
characterizes  our  times.   The  result  has  been  a  marked  loss  of  relative 
congressional  power  as  events  and  Executive  Branch  development  outstripped 
the  evolution  and  practice  of  oversight.   The  cure  for  this  imbalance  is 
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not  to  diminish  executive  Branch  capability  to  deal  with  the  volume  and 
complexity  of  information,  but  to  increase  congressional  capability 
to  deal  with  the  volume  and  complexity  of  information  it  needs  for  over- 
sight. 

New  Forces  Come  Into  Play  After  a  Law  is  Passed 

The  interests,  the  perspectives  and  the  pressures  which  coalesce  in 
Congress  to  raise  an  issue  or  to  pass  a  law  differ  from  those  which  operate 
in  the  Executive  Branch  after  the  law  is  passed.   Evaluation  within 
Executive  agencies  is  an  adjunct  to  good  management  and  should  be  en- 
couraged and  supported  despite  the  discomfort  it  sometimes  brings  to 
bureaucracy.   But  some  agencies  actively  resist  doing  internal  evaluation — 
and  are  even  less  likely  to  cooperate  actively  in  developing  evaluation 
evidence  for  congressional  overseers.   Even  when  an  agency  is  actively, 
genuinely  committed  to  internal  evaluation,  the  priorities  and  plans 
which  take  shape  within  the  Executive  do  not  necessarily  jive  with  those 
of  Congress.   Evidence  which  might  indicate  need  for  internal  improvement, 
or  need  for  actions  other  than  those  proposed  or  taken  by  the  agency 
for  other  reasons  is  unlikely  to  be  promptly  and  clearly  provided 
voluntarily  by  even  the  most  diligent  agency  evaluation  units.   Nonetheless 
practical  factors  (e.g.,  the  burden  of  data  collection)  preclude  complete 
duplication  of  evaluation  efforts  by  the  Executive  and  Congress.   Better 
access  to  data  for  Congress,  stronger  planning  for  oversight,  and  stronger 
technical  approaches  (e.g.,  independent  simultaneous  statistical  analyses 
of  shared  data)  can  do  much  to  alleviate  these  problems  however. 

Using  Evaluation  Constructively 

Evaluation  should  be  used  to  discover  what  things  work,  and  how 
successful  efforts  differ  from  similar  but  unsuccessful  efforts.   Data 
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produced  by  evaluation  should  be  used  to  sort  out  those  things  which  can 
be  accomplished  from  among  the  many  worthy  goals  and  efforts  which  compete 

for  scarce  resources.    Specific  evaluation  techniques  suitable  for  a 
variety  of  legislative  concerns  are  available.   But  none  of  these  specific 
techniques  can  or  should  substitute  for  traditional  congressional  investi- 
gation.  Evaluation  is  not  a  "scientific"  alternative  to  that  congressional 
or  journalistic  investigation  which  draws  attention  to  flaws  and  shortcomings 
to  mobilize  political  forces. 


Legislative  Precedent  for  Evaluation 

Congress  has  provided  the  authority  for  Executive  evaluation  in  a 
wide  variety  of  legislation  but  relatively  little  of  this  has  been  tailored 
to  oversight  needs.   Though  most  experience  and  legislative  authority  for 
evaluation  has  focused  on  Executive  Branch  management  usage  rather  than 
congressional  oversight  usage,  Congress  does  use  evaluation  on  an  occasional : 
ad  hoc  basis  (e.g.,  the  computer  simulation  of  alternative  funding  formulae 
for  ESEA  Title  I  done  by  C.R.S.).   Some  of  the  techniques  familiar  to 
Congress  (e.g.,  staff  investigation  teams,  polls  of  constituents)  are 
sufficiently  similar  to  special  evaluation  techniques  that  in  many  cases 
these  could  be  made  much  more  powerful  and  productive  than  they  already 
are. 

Key  Elements  of  Evaluation  for  Legislative  Users 

Evaluation  is  one  oversight  technique  among  a  whole  array  of  oversight 
techniques  which  Congress  has  at  hand.   Evaluation  is  not  a  substitute  for 
these  other  techniques.   Evaluation  (in  any  of  its  specific  types)  should  be 
coordinated  with  other  oversight  techniques  in  a  whole  oversight  strategy. 
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Used  tactically  within  an  oversight  strategy,  properly  selected  and  coordinated, 
evaluation  techniques  can  produce  a  flow  of  concise  information  which  strengthens 
other  oversight  techniques  and  is  valuable  in  itself. 

This  happy  outcome  will  not  result  merely  from  innate  characteristics 
of  evaluation  itself.   Successful  use  of  evaluation  in  legislative  processes 
requires  attention  and  control  by  evaluation  users.  User  control,  however, 
does  not  mean  making  technical  judgments  by  political  fiat,  but  rather 
properly  distinguishing  and  protecting  political  judgements  from  technical 
information.   User  control  and  attention  should  keep  the  evaluation  re- 
searcher targeted  on  the  substance  and  time-frame  of  his  client's  needs. 
It  must  not  be  exercised  to  censor  results. 

Evaluation  Resources  for  Congressional  Users 

Oversight  evaluation  can  originate  in  at  least  three  different  settings 
which  make  a  difference  in  access  to  evaluation  resources: 

-  in  the  history  or  language  of  a  particular  bill  or  act, 

-  within  the  work  agenda  of  a  particular  committee,  and 

-  in  an  individual  legislator's  personal  office. 

The  resources  available  vary  under  each  of  these  different  origins. 
In  the  history  or  language  of  an  act,  it  may  be  possible  for  Congress  to 
obtain  evaluation  resources  through  the  evaluation  staff  of  an  Executive 
Branch  agency.  The  various  congressional  agencies  (i.e.,  CRS,  CBO,  OTA, 
GAO)  and  services  (e.g.,  House  Information  System)  can  provide  resources 
needed  for  evaluation.  Availability  varies  with  the  agency  involved  and 
the  source  of  the  request  for  resources. 

When  working  in  an  individual  legislator's  office  more  resourcefulness 
is  required  to  obtain  evaluation  services.   But  it  can  be  done  well  and 
powerfully  if  the  office  is  willing  to  assign  a  staff  member  to  locate  and 
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monitor  the  necessary  professional  evaluation  resources  in  settings  out- 
side the  Congress  and  its  agencies.   Such  resources  can  often  be  obtained 
at  no  cost  or  low  cost  if  the  office  is  in  a  position  to  use  fellowship 
programs,  university  research  centers  or  faculty  or  other  concentrations 
of  personnel  trained  in  evaluation  research.   But  to  preserve  his  own 
control  there  must  be  good  user  liaison  and  monitoring  of  such  resources. 
Through  various  computer  index  and  access  systems  9  e.g.,  SCORPIO)  or  the 
quick-response  staff  at  Library  of  Congress,  some  evaluation  reports  can 
be  gotten  "off  the  shelf."  But  chances  are  that  these  reports  will  be 
lengthy,  mostly  irrelevant  and  out  of  date  for  your  concerns.  Such  faults 
are  not  with  the  retrieval  systems  which  are  regularly  updated,  but  are 
due  more  to  lack  of  adequate  user  monitoring  when  the  evaluation  was 
being  done. 

What  is  Takes  to  use  Evaluation  in  Oversight 

Using  evaluation  on  an  ad  hoc,  catch-as-catch-can  basis  can  have 

limited  but  dramatic  effects  occasionally.   But  this  approach  is  more 

mischievous  than  effective  in  gaining  substantial  changes.   If  you  wish 

to  use  evaluation  systematically  to  give  power  and  substance  to  oversight 

activities  you  will  need: 

o  Professional  staff  assigned  to  locate,  monitor  and  focus  evalua- 
tion resources  and  activities  and  coordinate  them  with  oversight 
activities; 

o  access-  to  professional  evaluations,  i.e.,  assistance  of  evaluation 
specialists  working  outside  the  congressional  staff  in  a  variety 
of  institutional  settings;  and 

o  data — either  raw  data  collected  specifically  from  the  activities 
being  overseen,  or  secondary  data  collected  originally  for  other 
purposes  but  reanalyzed  to  suit  oversight  needs  in  the  specific 
situation. 
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Coordinating  Evaluation  Techniques  and  Legislative  Processes 

Assuming  the  capacity  to  put  the  above  resources  together,  how  and  when 
can  they  be  used?  The  kind  of  evidence  needed  depends  on  what  the  particular 
oversight  question  is.   The  particular  evaluation  technique  to  be  invoked 
depends,  thus  on  the  kind  of  question  being  asked.   If  at  least  the  most 
important,  predictable  oversight  questions  are  framed  well  in  advance  it 
will  be  much  easier  to  provide  for  and  get  the  type  evaluation  evidence 
needed. 

When  you  choose  to  bring  that  evidence  into  play  is  a  matter  of  poli- 
tical judgement — although  the  natural  life  of  legislation  would  make  cer- 
tain questions  premature  or  irrelevant  at  certain  points. 

A  brief  guide  to  evaluation  techniques  which  provide  evidence  best 
suited  to  answer  some  general  types  of  oversight  questions  is  shown  in 
Table  1  (on  the  next  page).   Beside  each  question  are  listed  a  variety  of 
evaluation  techniques  which  can  generate  the  type  of  evidence  needed  to 
answer  that  question.   This  table  can  be  used  as  a  guide  to  the  kinds  of 
evaluation  that  should  be  called  for  in  legislative  language  or  history. 
Or,  it  can  be  used  in  an  ad  hoc  approach  to  assist  in  finding  the  most  rele- 
vant types  of  evaluation  for  the  type  evidence  you  need. 
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TABLF   1. 
EVALUATION  STRATEGIES  AND  TECHNIQUES  WHICH  KAY  HELP  TO  AKSWES  OVERSIGHT 


IV  lEt-ISLATIVE  ACTIVITIES. 


OVFFSICHT  questh 


EVALUATION   STRATECIES  ASH  ' 


/ 

1.    need,  assess.eot:   opinion   surveys:  Delphi   ,„,„,«. 

2.    What  kind,  at   talog,   »ul«  bo  <„.   «   re-*,   the  proM«, 

7.    policy    analysis:    evaluation    synthesis;    ,„for-at,on    retrieval    . 

3.    Mut    evidence    Is    there    to    Indicate    that    each   of    these    reredlcs   could        i 

,.    inforaa.lon   retrieval;    pilot   testing;   evaluation  «,„<„„,.. 

4-  earjfevjsir.ts.arrtfe ,.  jl*  *.  ....J 

-^-•l"1   8;    """•""'/benefit   analysis; 

s    (services)  directly?  To  fund  slate  or  local  aotwrma-cs 
to   develop  or   supply   remedies?     To  provide   incentives  for   the  private     1 

5.   policy  analysis:  PrMriCA,  ,:^OT 

6.    Which   combination  ol   reaedy  and   Tederal   role  best   suits   the 
goals  and    Intentions   of   a  Bill   or  Act? 

,    p„ot    testing;    planned  variation;    policy   analysis;   rei„«„  ,_ 

7.    Do   the  goals  and   Intentions  of  a   Bill   or  Act   catch   up  with   the 

nature  o(   the   ptoblca,    the   reaedles  and   the   Federal   role  envisioned 
by    the    Bill    or  Act? 

7.    policy    analysis;    POLITICAL   J0BQ 

6.    Does   the   Sill/Act   say  what   Confess   Intends   cjcjrlj   enough    (In   legl- 

to   set  criteria   against  which  you   can   ccopare   Congressional   Intent 
with   an    Ae^ncy's    pla-,s.    pr^in    l-pler.entatlon    and   budgets? 

8.    POLITICAL  JVDCW.EKT    In    consultation   with    Independent    evaluation   specialises 

S.    Are  Ajc^cy  goals   and  plans   cleat   enough     for  you   to   tell    11    they 
are  consistent  with  Coneresslcral   goals   and   Intents! 

,.    pre-.sse^nc   of   ev.luablHty;    certificate   of    t.adlne.s    for   evaluation 

iO.    Are  Ajency  prograa  plans  and  objectives  clear   enough   to  evaluate   Tor 
raecuej ve-Cranc.1)  policy   review  and  prograa  nanageaent?  What 

10.    pre-assessaent   of   ev.lu.b.Ut,;   policy   ,na,ys,.;    , 

11.    Are  program  services  being  delivered   in   the   type,   aacunt,    quality 
and   tlne-frane   that   Congress    intended? 

analysis   approaches);   prograa  auditing;    tine   and  iwotfcn  stu-JIes;   felue- 

What   kinds  of   projects  are  not  succeeding?  How  aany?    How  are 

12.    success   seaich;   bocus   operandi   evaluation;    institutional   self-study 

13.    Would   the   results   reported   for  a  prograa  have  happened  even  without 
the  prograa? 

13.    "experimental"    progran/evaluat Ion  design;    nodus   operandi    evaluation 

14.    Is   the  program  effective    In  meeting   the   goals   or  needs    set   foith 

U.    effectiveness   evaluation;    adversary  rval^Cm;    g^^^a      - 

)5.    Is   the  prograa  effective   cnou??)   to    solve   the   prcbleo   that    the 

15.    lapact  evaluation;    needs  assessment;   public   opinion   survey 

are   they? 

H.    slde-effects    .u.vey;    ..goal-free-    e.a on; 

17.    What   can  a  prograa  ecan   In  personal,   huaan  tei.sl   The  bad?   The  good? 

.7.    h„™„   sources   accounting;    „,.-,...„..:    .     .    ,..      „      ..,.,,«    ....,„ 

18.    Has   the   probleo   changed   that  was   originally   addressed  by   the   Act7 

:0.    Should  wc  continue   the   prograa?   As    It    Is?   with  najor   chanres? 

■JO  7 


Providing  Legislative  Authority  and  Direction  for  Evaluation 

To  plan  ahead  for  oversight  of  executive  activities,  the  necessary 

evaluation  provisions  should  be  written  into  the  language  or  history  of 

an  act.   The  more  conspicuously  this  is  done,  the  more  likely  that  the  right 

type  of  evidence  will  emerge  at  the  right  time.   And,  strong  evaluation 

provisions  often  have  the  desirable  side-effect  of  mobilizing  Executive 

activities  more  conscientiously  than  when  Congressional  follow-up  is  not 

expected. 

To  adequately  provide  for  evaluation  such  language  should  cover 

the  following  points: 

"purposes  of  the  evaluation  activities; 

°who  will  conduct  the  evaluation  research   (i.e.,  the  specific  or- 
ganizational unit  of  the  Executive  or  congressional  agency  chosen); 

°the  subject  matter  of  the  evaluation  (e.g.,  "measure  public  per- 
ception of  the  responsiveness  of  government  at  the  Federal,  State 
and  local  levels.  .  ."); 

"when  each  kind  of  evidence  from  the  evaluation  will  be  needed  (e.g., 
evidence  of  program  implementation  or  service  delivery  to  be  pro- 
vided yearly) ; 

"specific  funding  to  support  evaluation  activities  (if  clear  arrangements 
are  not  made  funds  for  evaluation  are  often  usurped  by  more  comfortable 
or  dramatic  activities); 

"reporting  arrangements  (to  whom,  when,  what  format);  and 

"access  to  data  (especially  important  to  secure  when  an  executive 
agency  will  be  conducting  the  evaluation,  if  you  wish  the  data 
analyzed  from  alternative  points  of  view) . 

Some  appropriate  phrases  covering  various  particulars  under  each  of 
these  points  may  be  found  in  Program  Evaluation:   A  Manual  for  Legislators 
and  Legislative  Staff.  If  all  the  above  points  are  covered  in  language 
and  history,  a  sound  basis  for  oversight  oriented  evaluation  will  be  pro- 
vided. 
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These  points  may  also  be  used  to  review  the  adequacy  of  oversight 
provisions  in  existing  legislation.   If  any  are  missing  it  may  be  diffi- 
cult to  oversee  the  legislation  in  other  than  a  catch-as-catch-can  basis. 
Broad,  permissive  provisions  for  evaluation  can  work,  but  will  require 
special  attention  if  they  are  to  actually  yield  timely,  relevant  evalua- 
tion evidence.   Overly  prescriptive  provisions  can  be  equally,  if  not 
more  troublesome — e.g.,  stipulating  specific  measures  or  statistical 
techniques  that  later  prove  susceptible  to  administrative  manipulation. 

Technical  Leverage  Points  to  Monitor  or  Review 

Unless  you  are  trained  and  experienced  in  doing  evaluation  research, 

to  become  too  involved  in  technical  details  of  evaluation  will  consume 

an  inordinate  amount  of  staff  time  and  may  disrupt  or  delay  the  necessary 

research  work.   However,  legislative  staff  can  and  should  have  a  general, 

critical  familiarity  with  the  substance  of  an  evaluation  on  the  following 

technical  leverage  points. 

-the  goals  or  needs  addressed  by  the  activities  being  evaluated  as 
defined  in  the  evaluation; 

-the  extent  to  which  evaluated  activities  are  actually  implemented 
or  delivered  as  described; 

-the  actual  measures  that  will  be  fed  into  the  evaluation; 

-the  logical  plan  for  collecting,  comparing  and  drawing  inferences 
from  the  data — i.e.,  the  research  design;  and 

-the  kinds  of  statistical  analyses  which  will  be  used  on  the 
data — there  is  usually  more  than  one  "right"  approach. 

Goals 

Evaluators  not  accustomed  to  working  in  legislative  settings  place 
great  emphasis  on  the  need  to  state  goals  clearly.   But  often  the  goals  of 
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legislation  are  broadly  stated  to  obtain  passage  of  the  legislation. 
This  need  not  and  should  not  prevent  evaluation.   An  evaluation  can  be  re- 
ferenced to  the  results  of  an  empirical  needs  assessment,  to  detailed  program 
plans  submitted  by  an  executive  agency  for  approval  of  an  oversight  committee, 
or  to  legislative  history  that  clearly  stipulates  the  reference  points 
intended. 

Service  delivery 

Evaluation  of  program  effectiveness  or  impact  is  unsuitable  without 
prior  evidence  to  establish  that  the  program  was  actually  delivered  as 
described.   Tactics  such  as  "certification  of  readiness  for  evaluation" 
will  help  prevent  the  error-  of  inferring  that  an  inadequately  implemented 
program  would  not  work  if  adequately  implemented. 

Measures 

Measures  should  be  pertinent  to  the  kinds  of  issues  Congress  was 
concerned  with  in  enacting  the  legislation,  as-  well  as  consistent  with  the 
goals,  needs,  or  program  descriptions  to  which  the  evaluation  is  referenced. 
Multiple  measures  should  be  used  wherever  feasible. 

Design 

The  logical  plan  for  collecting,  comparing  and  interpreting  the 
data  should  be  able  to  rule  out  or  minimize  alternative  explanations  for 
the  results  obtained.   "Experimentation"  evaluation  is  the  strongest 
technique  for  ruling  out  alternative  explanations  of  results. 

Alternative  explanations 

The  possibility  that  findings  yielded  by  an  evaluation  are  an  artifact 
of  a  particular  statistical  technique  can  be  minimized  by  conducting  al- 
ternative, simultaneous  statistical  analyses  of  the  same  data.   Results 
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from  such  competing  analyses  of  the  same  data  should  be  compared  for 
similarities  and  differences  of  findings  they  yield. 

Some  Cautions  For  Using  Evaluation 

The  Data  Burden.   Request  for  data  inundate  participants,  staff  and 
managers  of  Federal  programs  at  the  state  and  local  levels.   Responding  to 
such  requests  diverts  resources  required  for  program  operations.   Ordinarily 
the  data  compiled  do  not  benefit  the  program,  because  the  data  are  organized 
into  categories  that  are  used  for  some  Federal-level  administrative  pupose. 
The  expense  of  collecting  and  compiling  the  data  is  borne  at  the  local  level, 
ordinarily  without  any  Federal  support. 

Legislators  and  their  staff  contribute  to  such  burdensome  and  wasteful 
data  collection  in  two  ways.   First,  two  different  purposes  for  data  col- 
lection are  often  mixed: 

a.  to  identify  and  pursue  individual  cases  non-compliance, 
malfeasance  or  project  failure  in  order  to  stimulate  legal  action 
or  exert  pressure  on  behalf  of  individual  interests, 

b.  to  oversee  programs  or  jurisdictional  areas  (e.g.,  welfare 
benefits)  on  the  whole  for  the  purposes  of  effecting  program- 
matic improvement,  authorization,  appropriations,  or  gauging  the 
need  for  new  legislation. 

Second,  legislators  and  staff  are  prone  to  rely  far  too  heavily  on  one-number 
indices  of  program  implementation  or  effectiveness. 

The  pursuit  of  data  to  redress  abuses  in  individual  cases  should  be 
gathered  in  traditional,  casework,  investigatory  ways.   It  is  extremely 
wasteful  to  tie  up  national  data  collection  with  cops  and  robbers  tactics 
which  do  not  provide  adequate  data  for  consideration  of  broad  areas  of 
problems  and  performance.   Reliance  on  one-number  indices  usually  goes  hand- 
in-hand  with  this  approach.   The  result  is  a  proliferation  of  low-quality 
data  of  very  limited  use  for  broad  legislative  action. 
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Executive  agencies  must  also  share  some  of  the  blame  for  insufficient 
planning  of  data  needs  and  usage.   Failure  of  interagency  coordination 
may  cause  local  respondents  to  answer  the  same  type  question  with 
different  configurations  of  data  for  different  surveys. 

Evaluation  Failure  or  Program  Failure?   Past  experiences  (e.g.,  the 
evaluation  of  Headstart)  have  taught  that  flaws  in  evaluation  research 
design  should  be  eliminated  before  drawing  conclusions  about  the  success 
of  failure  of  that  which  is  evaluated.   Failure  to  establish  that  a 
program  was  implemented  as  described  or  premature  measurement  of  program 
effects  are  two  of  the  simpler  evaluation  failures.   Beyond  these  are 
the  more  esoteric,  but  important  problmes  associated  with  measurement, 
research  design  and  statistical  techniques. 

Evaluation  bias.  Evaluation  techniques  are  empirical  and  usually 
objective,  but  no;' unbiased .   Evaluation  biases  arise  from  necessary 
working  assumptions,  from  technical  judgements  that  must  be  called  and 
from  other  legitimate  sources.   Barring  the  unlikely  elimination  of 
biases,  it  is  wise  to  think  of  the  findings  of  evaluations  more  as  if 
they  were  legal  evidence  than  scientific  fact.   Law  has  developed  a 
variety  of  procedures  and  safeguards  for  dealing  with  evidence  that  can 
be  most  helpful  in  working  with  findings  from  evaluation.   The  applica- 
tion of  select  legal  concepts  to  use  of  evaluation  findings  is  being 
developed  by  Robert  L.  Wolf,  currently  at  University  of  Indiana. 
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DISCUSSION 

Several  points  about  priorities  for  doing  and  using  evaluation  were 
raised  in  the  discussion  following  the  Fox/Foskett  presentations.   It  was 
noted  that  frequently  political  considerations  and  political  expediency 
delimit  both  the  selection  of  programs  to  be  evaluated  and  the  use  of 
evaluation  information  in  decisionmaking.   The  use  of  evaluation  in  de- 
cisionmaking might  be  enhanced  if  congressional  committees  and  Federal 
agencies  developed  a  system  of  priorities  for  evaluation.   Such  priority 
statements  should  outline  specifically  the  questions  which  should  be 
answered  in  the  evaluation.   Additionally  the  quality  of  evaluation 
research  might  be  improved  if  simultaneous  evaluations  were  conducted 
of  the  program.   Similarly  it  was  noted  that  there  already  are  a  wealth 
of  evaluation  reports  and  secondary  analyses  of  many  programs.   Staff 
might  find  it  useful  to  review  these  reports  before  embarking  on  an 
evaluation  in  order  to  improve  the  formulation  of  questions  about  the 
program  and  to  enhance  cost/effectiveness  in  doing  evaluations. 
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"THE  STATE  OF  THE  ART  OF  PROGRAM  EVALUATION", 
CLARK  C.  ABT,  ABT  ASSOCIATES 


Ladies  and  Gentlemen,  my  topic  this  afternoon  is  the  state  of  the  art 
of  program  evaluation.   I'll  try  and  cover  as  briefly  as  I  can  something 
about  what  program  evaluation  can  do  and  can't  do,  particularly  from  the 
legislative  aspect,  how  the  program  evaluation  state  of  the  art  advanced 
in  the  last  decade,  who  can  do  program  evaluation,  what  we've  learned 
about  how  it  should  be  done,  how  quickly  and  cheaply  it  can  be  done, 
and  some  future  directions. 

First  of  all  in  terms  of  what  program  evaluation  can  do  and  what  it 
can't  do,  I  make  some  assumptions  of  rationality  that  more  information  is 
better  than  less  and  that  there  is  a  high  cost  to  uninformed  decision 
making.   These  are  assumptions;  they  are  not  always  operating  explicitly, 
but  ideally  would  be.   Cliff  Graves  mentioned  some  criteria  at  the  Office 
of  Management  and  Budget  in  terms  of  what  evaluations  should  be  able  to 
do  —  they  have  to  be  timely,  relevant,  and  accurate.   We  have  empirical 
evidence  that  evaluations  can  be  as  timely  as  two  weeks.   Recently  the 
Office  of  Technology  Assessment  operated  some  evaluations,  and  one  on 
the  ERDA  took  two  weeks,  another  one  on  offshore  exploration  which  had 
significant  impacts  on  leasing  policy  took  three  weeks.   So  the  ratio- 
nalization that  we  don't  have  time  to  do  evaluations  because  it  takes 
years  and  we  have  to  make  decisions  in  weeks  is  not  valid.   Evaluations 
can  be  done  in  weeks. 
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Years  ago,  I  used  to  be  involved  in  doing  technical  intelligence 
evaluations  of  large  scale  weapons  systems  and  we  usually  had  a  week  to 
respond  to  what  the  Air  Force  called  "quick  reaction  capability  require- 
ments". We  found  that  in  general  these  one-week  evaluations  of  new 
intelligence  information  on  systems  developed  about  90%  of  the  informa- 
tion about  program  effectiveness  that  we  got  in  a  further  year's  addi- 
tional study.   I'm  not  saying  that's  always  the  case  in  social  research, 
but  a  great  deal  can  be  done  in  a  very  short  period  of  time. 

In  terms  of  meeting  the  criteria  of  relevance,  the  question  is, 
"Can  program  evaluation  answer  questions  of  relative  benefits  and  costs, 
and  of  cost-effectiveness  and  impact  of  programs?"  I  believe  that  eval- 
uations can,  although  they  frequently  do  not,  and  I'll  get  into  some 
of  the  obstacles  to  their  answering  the  relevant  questions  in  a  few 
minutes. 

We  can  measure  cost-effectiveness  and  cost /benefit  of  any  kind  of 
program.   Let  me  draw  the  distinction  between  cost-effectiveness  and 
cost/benefit.   Cost  effectivness  is  easier;  some  criterion  of  effective- 
ness is  set,  such  as  advance  in  reading  scores  in  an  educational  pro- 
gram, and  the  amount  of  advance  per  unit  of  cost  is  the  cost-effectiveness 
or,  more  correctly,  the  effectiveness  divided  by  the  cost.   It's  basi- 
cally a  measure  of  the  desired  program  output  over  cost  input. 

Benefit/cost  is  harder  to  compute  because  here  we  are  computing 
benefits  and  costs  in  commensurate,  usually  dollar  terms.   We're  trying 
to  estimate  what  amounts  to  computation  of  return  on  investment,  so  that 
we  can  compare  different  programs.   It  is  hard  to  compare  years  of 
reading  achievement  per  dollar  with  percentage  reduction  in  cancer 
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fatalities  per  dollar  —  two  completely  different  programs  in  education 
and  health.   If  you  have  benefit/cost  ratios  for  both,  or  return  on 
fungible  resources  invested,  you  can  compare  those  and  when  you  can 
compare  across  programs,  ideally,  you  can  optimize  the  whole  portfolio 
of  government  investments  by  what  the  economists  call  equalizing  marginal 
utilities. 

All  programs  have  a  production  function  which  usually  shows  grad- 
ually increasing  productivity  per  investment  up  to  some  point  of  de- 
clining or  diminishing  returns  to  scale,  at  which  point  they  taper  off. 
Ideally  one  would  invest  in  all  the  programs  at  the  accelerating  part 
of  the  curve,  and  when  they  start  to  flatten  out,  disinvest  and  invest 
in  other  programs  that  are  still  accelerating  their  productivity.  One 
can't  do  that  unless  one  has  benefit/cost  measures.  We  do  know  how  to 
get  those  with  program  evaluation. 

We  can  also  operate  within  programs  to  determine  the  components 
that  are  strongest  and  weakest  and  thereby  improve  programs.  We  can 
evaluate  past  programs  through  summative  evaluations;  we  can  evaluate 
present  programs  through  formative  evaluations  in  which  continuous  feed- 
back is  provided  the  program  to  continue  to  improve  it;  and  we  can 
estimate  the  impacts  and  cost-effectiveness  and  benefit /cost  ratios 
of  future  programs,  either  with  simulation  or  if  there  are  too  many 
variables  involved  and  our  theory  is  inadequate  for  simulation,  through 
pilot  programs  and  planned  variation  experiments. 

We  can  determine  the  past,  present,  and  future  impacts  of  programs 
in  various  dimensions  of  concern  to  voters  and  policymakers  —  physical 
impact,  social  impact,  economic  impact,  political  impact,  impacts  on  the 
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actual  physical  state  of  the  environment,  on  the  state  of  people's 
opinions,  on  their  market  response  (we're  measuring  right  now  the 
elasticity  of  demand  and  supply  for  housing  under  conditions  of  housing 
allowances),  the  psychological  impacts,  the  behavioral  impacts. 

We  can  do  all  of  this  to  resolve  conflicts  of  program  choice, 
to  make  decisions  more  rationally  about  whether  to  start  up  a  program, 
whether  to  modify,  expand,  or  contract  a  program,  or  whether  to  terminate 
a  program,  and  we  can  aid  that  same  set  of  decisions  within  a  program 
about  program  components. 

What  we  cannot  do  with  program  evaluation  is  create  new  programs, 
make  decisions,  make  the  assumption  of  rationality  come  true,  and  get 
users  or  consumers  of  program  evaluations  to  ask  the  right  questions. 
We  also  cannot  make  value  judgments  that  have  any  credibility  where 
these  are  decisive  program  decisions. 

The  most  important  limitation  of  program  evaluation  is  that  we 
can't  get  people  to  use  it  —  at  least  not  consistently  —  and  there 
are  three  main  reasons  why  we  can't.  Keith  Marvin  mentioned  that  in 
a  survey  of  government  agencies  of  why  there  wa6  little  or  no  benefit/ 
cost  analysis  in  budgeting,  the  answers  were  that  it's  hard  to  measure 
benefits  and  costs,  there  is  not  enough  staff,  and  there  is  not  enough 
time.   I'll  mention  something  about  the  staff  and  time  resources  later. 
Concerning  it  being  hard  to  measure  benefits  and  costs,  I  think  there 
is  plenty  of  economic  and  other  theory  that  enables  us  to  measure  the 
benefits  and  costs  of  any  program.  The  next  major  breakthrough  in  pro- 
gram evaluation  will  not  be  in  methodology,  I  believe,  because  we  have 
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sufficient  methodology  to  do  the  job  now,  but  rather  in  government 
utilization  —  particularly  in  the  cross-cutting  agencies  of  0MB,  GAO 
and  Alice  Rivlin's  new  organization. 

The  three  main  obstacles  to  actual  program  evaluation  use,  I  be- 
lieve, are  goal  conflicts,  communications  failures,  and  knowledge  gaps. 
On  goal  conflicts,  part  of  the  problem  is  that  there  is  not  a  clear 
agreement  about  the  purposes  of  evaluation  between  the  sponsors,  the 
operators,  and  the  users.   For  example,  HEW  may  sponsor  an  evaluation 
that  Congress  will  use  and  that  some  independent  research  organization 
will  actually  operate.   They  don't  necessarily  all  have  the  same  in- 
terests, and  in  fact  there  is  a  functional  tendency  to  look  at  it  dif- 
ferently.  Evaluators  want  clarified  goals  and  clarified  consequences. 
Political  actors  who  sponsor  evaluation,  and  particularly  who  sponsor 
programs  to  be  evaluated,  sometimes  need  to  reduce  the  distinctions 
and  to  muddy  them  in  order  to  build  winning  coalitions  that  will  get 
programs  through.   So  they  may  not  have  the  same  interests,  at  least  not 
at  the  same  time,  as  evaluators. 

There  are  many  communications  failures  among  those  three  groups, 
partly  because  they  don't  always  speak  the  same  language.   I  won't  be- 
labor that  whole  issue,  but  I  think  the  main  reason  is  because  there 
are  unclear  statements  of  specifications  for  the  operators  of  program 
evaluations.   I  have  some  specific  suggestions  which,  if  they  were 
worked  into  legislation  requiring  program  evaluation,  might  reduce  the 
communication  failures  between  program  evaluators  and  program  evaluation 
users. 
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Most  frequently  the  work  statements  of  program  evaluation  requests 
for  proposals  describe  what  they  want  done  and  how  but  not  why,  and  there 
is  very  little  information  to  the  bidders  about  the  actual  purposes  of 
the  evaluation,  partly  for  the  very  realistic  reasons  that  purposes 
differ  and  that  might  be  a  source  of  conflict.  With  that  overspeci- 
fication  of  the  means  of  execution  of  program  evaluations,  we  have 
support  for  non-competent  bidders  because  they  have  their  work  laid  out 
for  them,  and  a  great  constraint  and  disincentive  to  innovation  on  the 
part  of  competent  bidders. 

On  the  other  hand,  some  work  statements  are  so  vague  and  some 
mandates  for  program  evaluation  are  so  global  and  unspecific  that  one 
gets  involved  in  a  guessing  game  of  what  is  really  intended  and  the  most 
persuasive  salesmen  win  the  work  rather  than  the  people  who  might  be 
the  most  competent.   In  terms  of  the  overspecification  error,  the  counter- 
measure  to  that,  I  believe,  is  limitation  of  the  specifications  to  out- 
puts of  program  evaluation  and  constraints  when  obtaining  those  outputs, 
without  definition  of  populations  to  be  sampled,  sample  sizes,  research 
design,  and  so  forth,  which  are  best  left  to  the  program  evaluators  to 
work  out. 

With  respect  to  the  error  of  underspecification,  the  trouble  here 
is  that  there  is  often  a  confounding  of  competence  testing  of  potential 
program  evaluators  with  contractual  definition  of  the  work  output.   It 
is  desirable  here  to  separate  the  two  and  make  competence  testing  of  the 
potential  program  evaluators  a  completely  separate  part  from  the  actual 
specification  of  the  work.  That  can  be  done  by  giving  an  exam  or  re- 
viewing records  or  asking  for  a  specific  competence  demonstration,  and 
then  separating  that  from  the  actual  work  to  be  done. 
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Work,  outputs  should  be  defined  in  operational  terms.   Most  of  the 
enabling  legislation  for  program  evaluation  that  I  have  seen  is  extremely 
vague  about  the  very  specific  questions  it  wants  answered,  and  here  I  think 
the  work  of  Bill  Foskett  and  Harrison  Fox  57/  is  quite  useful  in  helping  to 
give  you  an  idea  of  the  kinds  of  questions  that  need  to  be  specified.   It's 
very  hard  to  evaluate  evaluations  for  whether  they've  been  relevant  and 
answered  the  questions  if  one  doesn't  know  what  the  questions  are.   So  le- 
gislation calling  for  program  evaluation  should  have  very  explicit  statements 
of  what  the  minimum  set  of  policy  questions  and  program  questions  to  be 
answered  should  be. 

Now  I  make  a  distinction  between  asking  program  evaluation  questions 
and  insisting  that  all  programs  have  specified  goals.   Joe  Wholey  at  the 
Urban  Institute  insists  that  you  can't  evaluate  a  program  if  you  don't  know 
what  its  goals  are.   I  think  he's  certainly  correct  in  that  it's  much  easier 
and  better  evaluating  programs  whose  goals  are  specified,  than  where  goals 
have  changed  over  time,  or  their  goals  are  bogus  goals  that  mask  a  loose 
coalition  of  a  variety  of  goals  that  are  actually  operating.   So  if  one  re- 
quires a  clear  specification  of  goals  before  program  evaluation  can  be  effec- 
tive, one  immediately  opts  out  of  most  program  evaluations. 

There  is  no  requirement,  however,  that  we  can't  have  specific  ob- 
jectives for  program  evaluation,  even  if  we  can't  detect  specific  pro- 
gram goals.   In  other  words,  one  can  see  a  program,  let's  say  a  health 
maintenance  program  in  a  rural  community,  and  it  may  never  have  bothered 
to  put  its  goals  together  and  one  can  conceive  of  various  goals  that  are 
desirable  and  say,  now  the  object  of  the  evaluation  of  this  program  is  to 
determine  whether  this  rural  health  program  made  any  difference  to 
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Work  outputs  should  be  defined  In  operational  terms.   Most  of  the 
enabling  legislation  for  program  evaluation  that  I  have  seen  is  extremely 
vague  about  the  very  specific  questions  it  wants  answered,  and  here  I 
think  the  work  of  Bill  Foskett  and  Harrison  Fox  is  quite  useful  in  help- 
ing to  give  you  an  idea  of  the  kinds  of  questions  that  need  to  be 
specified.   It's  very  hard  to  evaluate  evaluations  for  whether  they've 
been  relevant  and  answered  the  questions  if  one  doesn't  know  what  the 
questions  are.   So  legislation  calling  for  program  evaluation  should 
have  very  explicit  statements  of  what  the  minimum  set  of  policy  questions 
and  program  questions  to  be  answered  should  be. 

Now  I  make  a  distinction  between  asking  program  evaluation  questions 
and  insisting  that  all  programs  have  specified  goals.   Joe  Wholey  at  the 
Urban  Institute  insists  that  you  can't  evaluate  a  program  if  you  don't 
know  what  its  goals  are.   I  think  he's  certainly  correct  in  that  it's 
much  easier  and  better  evaluating  programs  whose  goals  are  specified, 
than  where  goals  have  changed  over  time,  or  their  goals  are  bogus  goals 
that  mask  a  loose  coalition  of  a  variety  of  goals  that  are  actually 
operating.   So  if  one  requires  a  clear  specification  of  goals  before 
program  evaluation  can  be  effective,  one  immediately  opts  out  of  most 
program  evaluations. 

There  is  no  requirement,  however,  that  we  can't  have  specific  ob- 
jectives for  program  evaluation,  even  if  we  can't  detect  specific  pro- 
gram goals.  In  other  words,  one  can  see  a  program,  let's  say  a  health 
maintenance  program  in  a  rural  community,  and  it  may  never  have  bothered 
to  put  its  goals  together  and  one  can  conceive  of  various  goals  that  are 
desirable  and  say,  now  the  object  of  the  evaluation  of  this  program  is 
to  determine  whether  this  rural  health,  program  made  any  difference  to 
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the  health  of  the  rural  area  resident  —  whether  that  was  the  program 
goal  or  not,  it  should  have  been. 

It's  very  possible  to  write  evaluation  goals  into  enabling  legis- 
lation without  requiring  program  goals.   The  incomplete  or  absent 
explanation  of  why  the  research  is  needed  deprives  program  evaluators 
of  the  political  context  and  comprehension  of  what  questions  are  to  be 
addressed.   In  case  you  missed  putting  down  what  the  evaluation  objec- 
tives should  be  by  informing  the  program  evaluators  why  the  research  is 
needed,  they  can  at  least  then  derive  the  evaluation  objectives.   So 
its  very  important  to  put  those  down  to  add  to  general  context. 

Most  evaluation  results  do  not  feed  into  program  or  budget  deci- 
sions.  The  connection  is  mainly  random  right  now.   I  have  it  from 
some  highly-placed  authorities  who  I  am  sure  would  not  wish  to  be  quoted, 
that  there  is  no  significant  connection  now,  no  planned  connection  be- 
tween outputs  of  program  evaluations  and  the  major  programmatic  budgetary 
decisions  about  programs.   It  doesn't  mean  it  never  happens  —  I'm 
just  saying  it  doesn't  happen  usually  and  it  doesn't  happen  on  a  planned 
basis.   I  think  Cliff  Graves  pointed  out,  quite  accurately,  that  program 
evaluations  are  often  used  in  making  program  decisions;  but  I  think  we 
could  argue  that  they  should  always  be  used  and  even  if  they  are  used, 
the  results  may  be  a  little  stale  and  if  we  had  planned  for  the  output 
of  program  evaluation  to  feed  into  program  decisions,  the  results  would 
be  better. 

I  would  suggest  that  in  any  enabling  legislation  for  program  evalu- 
ations these  be  designed  as  decision-driven  evaluations  —  that  is,  the 
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scheduling  of  the  program  evaluation  results  should  allow  sufficient 
time  afterwards  for  consideration  for  a  decision  that  has  to  be  made 
using  those  results.  Many  program  evaluations  have  conflicting  goals, 
adding  to  the  communications  problem.  These  conflicting  goals  are 
reflected  in  either  global  work  statements  and  specifications  or  very 
fuzzy  ones.   A  way  of  dealing  with  this  legislatively  is  to  resolve 
the  goal  conflicts  internally  before  writing  the  specifications  for 
the  evaluation,  rather  than  expressing  conflicting  aims  in  the  speci- 
fication for  the  work.   If  these  conflicting  aims  cannot  be  resolved  — 
and  they  often  can't  be  —  it  is  better  to  accept  the  conflict  and  say 
we  will  simply  sponsor  two  kinds  of  evaluations  that  have  conflicting 
objectives  and  they  will  be  mutually  independent  and  we  will  resolve 
conflict  after  we  get  the  results  of  the  evaluations,  rather  than  to 
anticipate  that  conclusion,  mush  the  issue,  and  try  to  get  a  compromise 
evaluation  going. 

The  knowledge  gap  is  the  third  major  reason  for  program  evaluation 
being  underutilized,  the  first  two  being  goal  conflict  and  communication 
failures.  The  main  problem  in  the  knowledge  gap  is  that  it's  a  gap 
existing  on  both  sides,  both  on  the  side  of  the  program  evaluation 
sponsors  and  users  —  not  necessarily  the  same  group  —  and  the  evalua- 
tion researchers  or  producers,  and  neither  group  completely  or  fully 
appreciates  the  position  of  the  other. 

Program  evaluators  and  researchers  rarely  appreciate  the  full 
political  context  and  the  pressures  acting  on  government  decision  makers 
to  make  a  decision,  and  the  kinds  of  sensitivities  they  have  to  respond 
to.   On  the  other  hand,  government  decision  makers,  unless  they  come  from 
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a  background  of  social  science  or  evaluation  research,  rarely  appreciate 
the  possibilities  and  limitations  of  program  evaluation  research.  As 
a  result,  the  government  sponsors  generally  ask  for  less  than  they 
can  get  in  terms  of  speed,  relevance,  and  validity,  and  don't  insist 
on  as  high  standards  as  they  could  insist  on.   The  evaluators,  in  turn, 
are  not  as  responsive  and  communicative  with  respect  to  the  political 
needs  of  the  government  sponsors  as  they  could  be  if  they  had  the  infor- 
mation that  they  need  to  be  that  way. 

One  of  the  problems  is  that  evaluators  often  live  in  a  fantasy 
world  of  ideal  efficiency.   Cost-effectiveness  and  benefit/cost  ratios 
are  very  important  to  them,  and  after  one  has  been  working  very  hard 
to  get  these  things  for  many  years,  one  tends  to  assume  that's  what  the 
world  really  cares  about.   Then  it  comes  as  a  great  shock  after  years 
of  research  when  one  actually  talks  to  a  Congressman  or  a  legislative 
aide  and  starts  talking  efficiency,  and  the  other  person  seems  to  be 
backing  away  all  the  time  and  gets  a  distant  look  and  the  eyes  kind  of 
glaze  over,  and  you  know  you're  not  communicating.   Then,  if  you  have 
enough  time  and  you're  a  little  sensitive  and  you  draw  the  other  in- 
dividual out,  you  find  out  that  the  concerns  are  really  about  what 
Harold  Lasswell  very  ably  defined  years  ago  as  the  work  of  politics  — 
who  gets  what,  where,  when,  and  how,  and  not  how  efficiently  and  not 
how  effectively.  58/ 


sa/  Lasswell,  Harold  D.   Politics:  who  gets  what,  when,  how.   New 
York,  McGraw-Hill  Book  Co.,  1936. 


314 


Efficiency  and  effectiveness  are  considerations,  but  probably  not 
the  most  salient  considerations.   They  are  used  as  levers  or  sticks  to 
boost  or  depress  various  popular  or  unpopular  programs,  but  they  are 
not  the  central  issue  they  seem  to  be  the  methodologists.   So  there 
isn't  really  a  good  mutual  understanding  here  of  the  needs  of  evaluation 
research,  and  I  think  a  lot  of  that  can  be  addressed  by  simply  more 
communication  between  potential  users  of  the  evaluation  research  and 
the  evaluation  researchers  themselves. 

Often  there  is  the  sponsor  who  intervenes,  and  the  sponsor  may  not 
have  the  user's  best  interests  at  heart.   For  example,  the  user  may  be 
the  Office  of  Management  and  Budget  to  decide  program  allocations,  the 
sponsor  may  be  a  sub-department  of  an  agency  that's  trying  to  prove 
that  a  particular  program  is  terribly  cost  effective  in  order  to  con- 
tinue and  expand  it  indefinitely.   Obviously  here  the  user  and  the  spon- 
sor have  very  different  objectives.   The  sponsor  has  a  —  public  relations 
is  too  crude  a  term  —  a  support  objective.   The  user  has  a  comparison 
objective  —  these  are  very  different  goals  for  evaluation  research.   If 
the  evaluator  is  too  responsive  to  sponsors,  he  may  defeat  the  users' 
purpose.  All  three  groups  have  to  interact  in  the  process. 

How  has  the  program  evaluation  art  advanced  in  the  last  decade? 
First  let  me  suggest  that  it's  advanced  enormously  but  there  have  been 
hardly  any  theoretical  breakthroughs.   We've  had  an  engineering  advance 
but  not  a  scientific  advance.   Take  the  analogy  of  nuclear  weapons  or 
nuclear  power.   The  theoretical  work  on  that  was  pretty  much  completed 
in  the  30 *s  but  it  took  another  20  years  and  it's  still  taking  time  to 
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fully  realize  the  engineering  applications  of  that  theoretical  work. 
In  the  same  way,  most  of  the  statistical  concepts,  the  experimental 
designs,  the  instrument  design  issues,  the  operational  designs  of  pro- 
gram evaluations  were  done  in  the  1940 's  or  earlier  —  many  of  them  in 
World  War  II  operations  research,  in  earlier  public  opinion  and  mar- 
keting research,  in  survey  research  in  the  1930 's  developed  for  com- 
mercial purposes. 

The  scientific  work  was  done  a  long  time  ago,  but  in  the  last  ten 
years,  partly  through  the  diffusion  of  the  cost-effectiveness  analysts 
from  the  Defense  Department  to  the  social  programs  agencies  of  the 
government  since  the  middle  1960's,  these  methods  have  diffused  through 
the  rest  of  the  government,  and,  with  the  increasing  investment  in  social 
programs,  the  techniques  have  begun  to  be  applied.   A  lot  of  the  mistakes 
and  errors  we've  made  in  social  program  evaluations  have  resulted  in 
the  rediscovery  of  things  that  have  been  known  scientifically  for  20 
years  but  were  simply  not  known  to  social  programs  evaluators. 

A  good  example  is  the  regression  artifacts  that  limited  the  validity 
of  the  Westinghouse/Ohio  evaluation  of  Headstart,  which  as  you  may  know 
could  not  find  any  significant  achievement  gains  in  children  as  a  result 
of  Headstart,  and  had  enormous  policy  impact  in  probably  limiting  the 
funding  in  Headstart,  resulting  in  other  programs  to  maintain  the  effect 
that  extinguished  too  rapidly  such  as  the  Follow-Through  program,  and 
experimental  planned  variation  programs,  rather  than  massive  national 
programs  based  on  theoretical  assumptions  that  were  unproven  empirically. 
One  of  its  main  effects  was  the  methodological  one  showing  very  dramat- 
ically the  great  limitations  that  would  result  from  a  flawed  experi- 
mental design;  namely  the  lack  of  randomized  selection  of  experimental 
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and  control  groups  partly  responsible  for  the  regression  artifacts, 
which  made  it  impossible  to  attribute  the  findings  reliably  to  the 
program. 

We've  had  a  great  deal  of  technological  development  in  program 
evaluation  research  in  the  last  ten  years,  with  the  diffusion  of  sur- 
vey research  methods,  experimental  design  methods,  instrumentation 
methods,  diffusion  of  statistical  sophistication,  diffusion  of  cost/ 
benefit  and  cost-effectiveness  analyses  to  new  applications  from  their 
original  hardware  technology  and  military  applications  to  health, 
education,  welfare,  housing,  criminal  justice  and  so  on,  and  this  is 
still  going  on.  We've  had  an  increase  in  the  sophistication  and  sen- 
sitivity to  the  limitations. 

The  evaluation  art  has  advanced  to  the  point  today  where  we  can 
evaluate  anything,  we  can  determine  benefit/cost  ratios,  we  can  de- 
termine relative  and  absolute  cost-effectiveness  —  what  we  can't  do 
is  always  get  it  and  communicate  our  results. 

Another  major  change  in  the  last  ten  years  has  been  the  widespread 
diffusion  of  large-scale  data  processing  capabilities.  Ten  or  fifteen 
years  ago  a  lot  of  the  survey  research  we  carry  out  now  was  practically 
too  expensive.   In  our  housing  allowance  demand  experiment  we  did  a 
base-line  survey  of  100,000  to  finally  select  4,000  households  for  the 
randomly  selected  control  group.   Even  with  4,000  households  because 
there  are  quite  a  few  variables,  some  of  the  cells  just  barely  achieve 
statistical  significance  in  size.  This  was  done  for  a  few  million 
dollars  within  a  few  months.   Before  large-scale  data  processing  facil- 
ities were  available,  it  would  have  taken  many  years  and  probably  tens 
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of  millions  of  dollars  —  it  would  have  been  at  least  an  order  of 
magnitude  more  expensive.   So  we  have  enormously  greater  data  collec- 
tion and  data  processing  and  reduction  power  at  our  disposal  than  we've 
had  before,  and  we  are  not  using  all  of  it  that  we  have. 

Given  these  advances  in  the  state  of  the  art  in  the  last  decade 
and  what  program  evaluation  can  do  and  what  it  can't  do,  I'd  like  to 
quickly  mention  some  requirements  for  doing  it  effectively  in  tele- 
graphic form.   I  just  want  to  give  you  an  idea  of  the  richness  and 
complexity  of  this  field,  and  that  there  is  just  so  much  to  be  known 
and  used  here  beyond  what  has  been  used. 

The  evaluation  resources  of  competent  people,  money  and  time  are 
very  scattered,  and  one  of  the  things  we  need  to  do  is  concentrate  them. 
Instead  of  a  little  bit  of  evaluation  everywhere,  we  need  to  do  enough 
evaluation  to  be  timely,  relevant,  and  valid  and  accurate  for  the  big 
payoff  items.   The  highest  cost  programs  do  not  get  proportionately 
concentrated  evaluation  now;  we  need  to  allocate  in  proportion  to  impact. 
We  can  define  impact  in  a  variety  of  ways  —  but  generally  it's  func- 
tion of  how  many  people  are  affected,  how  much,  for  how  long,  how  inten- 
sively. We  generally  know  that,  other  things  equal,  programs  that 
affect  lots  of  people  have  more  impact  than  programs  that  affect  just 
a  few  people.   Programs  that  affect  people  deeply,  like  whether  it'll 
save  their  lives  or  not,  have  more  impact  than  those  that  are  somewhat 
more  aesthetic  or  cosmetic,  etc. 

Evaluations  often  compare  programs  within  agencies,  but  rarely  do 
they  compare  programs  with  similar  goals  in  different  departments.  We 
have  about  a  dozen  income  transfer  programs,  and  for  each  of  those  we 
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have  eligibility  and  certification  processes  which  have  never  really 
been  compared  across  different  programs.   Government  income  transfer 
programs  on  the  average  have  about  100%  overhead  —  if  FAP  had  gone 
through,  of  the  $4  billion  cost,  about  $2  billion  of  it  would  have  been 
just  for  administering  the  program.  A  large  percentage  of  that  is  for 
outreach  and  certification.  All  these  different  income  transfer  pro- 
grams do  certification  differently.  There's  never  a  cross-cutting  study 
to  analyze  which  agencies  do  it  best  and  most  cost-effectively  and  what 
they  can  learn  from  each  other. 

So  we  need  cross-cutting  evaluations  of  processes  and  functions 
that  are  common  to  a  great  many  different  agencies  programs.   I  think 
this  is  a  major  new  role  for  the  cross-agency  missions  of  the  Office 
of  Management  and  Budget,  General  Accounting  Office,  Alice  Rivlin's 
Office  59  /  and  the  Congress.   I  don't  see  how  this  can  be  done  com- 
pletely within  the  departments  themselves. 

Evaluations  are  rarely  scheduled  inputs  to  budget  decisions,  so  we 
advocate  decision-driven  program  evaluations  by  legislation.  Evalua- 
tions are  often  not  focused  on  controllable  policy  variables,  so  that 
the  evaluation  results  don't  always  help  in  budget  decision  making.  By 
a  policy  variable  I  mean  a  variable  you  can  do  something  about,  such  as 
the  particular  characteristics  of  a* program,  as  opposed  to  a  contextual 
variable,  something  you  can't  do  anything  about,  such  as  the  percentage 
of  people  under  the  age  of  30  living  in  a  town  —  you  can't  easily 
change  that.   A  lot  of  evaluation  research  for  academic  reasons  tends  to 
focus  on  contextual  variables  which  are  not  under  the  control  of  policy 


59/  The  Congressional  Budget  Office,  created  pursuant  to  P.L.  93-344. 
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makers,  instead  of  focusing  on  policy  variables  that  are.   These  could 
be  legislatively  specified. 

Evaluation  sponsors  and  researchers  rarely  communicate  directly 
with  the  most  important  consumers  of  evaluation.   In  about  20  years  of 
evaluation  research  I've  never  had  the  opportunity  to  discuss  the  pur- 
pose of  an  evaluation  with  the  congressional  leaders  or  legislative 
assistants  who  are  mostly  concerned  with  the  results  of  that  evaluation 
—  never. 

Some  evaluation  research  is  mi6designed  technically.   If  it  does 
employ  experimental  design,  there  is  often  no  randomized  selection  of 
experimental  and  control  groups,  and  thus  one  is  never  certain  that 
the  findings  are  not  the  results  of  a  confounding  of  some  external 
event  or  factor  with  the  program  that  is  supposed  to  produce  those 
results. 

There's  a  lack  of  any  control  groups  in  too  many  evaluation  designs, 
so  that  there  is  no  assurance  of  what  we  call  internal  validity,  that 
is,  the  attribution  of  effects  to  the  program  that  is  being  evaluated, 
or  external  validity,  the  generalizability  of  the  findings.   There's 
also  often  a  failure  to  recognize  and  compensate  for  the  statistical 
artifacts  that  I  mentioned  before,  such  as  regression  towards  the  mean. 

There's  a  frequent  failure  to  select  the  readily  measurable  measures 
of  effectiveness,  and  this  can  be  remedied  by  legislation  to  some  extent 
suggesting  these.   This  is  a  delicate  point,  not  specifying  but  sug- 
gesting some  of  the  measures  of  effectiveness  that  are  of  interest.   I 
say  suggest  rather  than  specify,  because  again  here  the  researchers  may 
come  up  with  more  useful  measures  and  may  find  it  unproductive  to  be 
constrained  to  specific  measures.   We  talked  about  the  too  broad  focus 
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which  can  be  corrected  by  legislation  specifying  the  questions  to  be 
answered,  too  narrow  focus  which  can  be  remedied  by  the  enabling 
legislation  not  specifying  methods  but  rather  results  desired  and 
questions  to  be  answered. 

Too  short  duration.   I  realize  I  talked  about  some  evaluations 
being  possible  within  two  weeks,  and  yet  some  evaluations,  particularly 
the  kind  that  depend  on  developmental  changes  in  human  beings,  may 
require  longitudinal  evaluations  lasting  years,  because  people  just 
don't  change  that  fast.   And  cross-sectional  evaluations,  that  is, 
the  instant  snapshot  comparing  many  things  at  the  same  time,  simply 
will  not  get  at  the  change  in  people  over  time  with  much  reliability 
and  validity.   It  is  possible  to  simulate  a  longitudinal  study  with 
overlapping  cross  sections  in  time,  but  it's  always  much  more  complex 
and  expensive  and  less  reliable.   So  if  we're  talking  about  the  im- 
pact of  daycare  children  or  the  impact  of  an  educational  program  on 
life  opportunity,  we  really  ought  to  have  the  patience  to  wait  and 
have  sufficient  duration  to  measure  the  more  subtle  effects. 

The  misdesign  of  instruments:   usually  through  compression  in  time, 
a  lack  of  pilot  testing,  and  corrective  feedback.  Any  commercial  sur- 
vey research  outfit  usually  has  time  to  pilot  test  and  check  its  survey 
instruments  —  this  is  not  always  possible  if  you  insist  on  doing 
evaluation  research  within  the  annual  budget  cycle.   Data  gathering 
misdesign  is  also  a  problem,  including  inappropriate  allocation  to  sur- 
vey resources,  sometimes  resulting  in  oversampling  of  the  least  sig- 
nificant groups  and  under  sampling  of  the  most  significant  groups.   I 
draw  your  attention  here  to  the  Watts-Conlisk  design  applied  to  the 
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New  Jersey  Income  maintenance  experiment.   This  was  a  good  piece  of 
research,  but  has  some  limitations  in  this  respect  because  most  of  the 
sampling  was  done  of  the  population  that  was  of  minimum  policy  interest, 
rather  than  the  reverse,  because  this  particular  sampling  allocation 
model  economized  in  this  particular  direction. 

I  think  the  only  kind  of  regulation  we  can  look  for  here  is  self 
regulation  in  the  field  itself.   Qualitative  competition  among  the 
evaluators  themselves  could  really  deter  faulty  work  in  the  field. 
For  a  government  policy  on  this,  I  would  relate  awards  and  fees  to  a 
consumer's  report  on  the  quality  of  previous  work.   That  always  raises 
administrative  problems  about  who  is  to  judge,  etc .   Obviously,  it  is 
not  in  the  interest  of  a  lot  of  sponsoring  agencies  to  want  this,  so 
we  cannot  count  on  them.   But  I  think  there  are  some  other  ways  besides 
an  internal  regulation. 

On  the  overall  problem  of  the  knowledge  gap,  I  think  it  is  very 
hard  to  ever  really  appreciate  the  capabilities  and  limitations  of 
evaluation  from  the  outside  without  having  done  it.   It  is  a  kind  of 
way  of  looking  at  things.   I  think  it  is  unlikely  that  the  intended 
users  in  the  legislature  and  in  the  executive  branch  will  really  ap- 
preciate it  until  they  get  involved  in  it.   How  can  we  tempt  them  to  get 
involved  in  it?  This  is  a  paradox,  because  as  has  been  pointed  out 
by  Professor  Haveman  607  and  by  Senator  Brock,  they  are  not  particularly 
eager  to  have  their  own  activities  and  their  own  processes  evaluated. 


fin/  Professor  Robert  H.  Haveman,  University  of  Wisconsin. 
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I  think  the  next  best  thing  is  to  get  them  to  participate  in  some  of 
the  planning  of  evaluations.   That  gives  a  sense  of  control,  a  sense 
of  power  over  things,  and  basically  more  sharing  by  the  executive  and 
the  administration  with  the  legislative  branch  of  the  actual  setting 
up  of  program  evaluations.   It  is  still  a  sequential  process  now  rather 
than  an  integrated  process.   I  don't  know  how  to  do  this,  but  I  think 
that  some  of  this  can  be  done  informally  by  the  practitioners  themselves. 

Getting  back  to  the  legislative  sponsor  of  the  evaluation,  which 
is  often  very  much  frowned  upon  by  the  executives  in  the  administra- 
tion, involves  some  risk  which  has  to  be  taken  here  of  informal  con- 
tacts asking  "Is  this  really  what  you  want  to  find  out?" 

In  summary,  I'd  just  like  to  describe  some  particular  weaknesses 
of  evaluation  that  we've  noticed  and  that  I  think  if  reduced  or  re- 
moved could  improve  the  usefulness  of  evaluation  and  applicability  of 
the  work  to  legislation. 

Some  of  these  typical  weaknesses,  in  no  particular  order,  are: 

Lack  of  randomization  in  the  selection  of  experimental  and 
control  groups,  biasing  the  findings. 

Lack  of  any  control  groups,  rendering  the  validity  and 
generalizability  of  the  findings  completely  uncertain. 

Failure  to  recognize  and  compensate  for  statistical  arti- 
facts such  as  regression  towards  the  mean. 

Failure  to  select  the  readily  measurable  measures  of  effec- 
tiveness relevant  to  the  policy  issues. 

Too  broad  a  focus,  and  failure  to  limit  the  evaluation  to 
program  aspects  that  can  be  researched. in  enough  depth  to  achieve 
validity  within  the  time  and  effort  resources  available  —  the 
"fising  expedition"  that  generates  a  low  signal-to-noise  ratio. 
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Too  narrow  focus,  with  premature  concentration  on  a  few 
variables  assumed  to  be  significant  at  the  cost  of  exclusion 
of  potentially  more  significant  ones. 

Too  short  duration  to  allow  sufficient  iteration  of  the  data 
gathering  efforts  to  narrow  down  to  the  critical  issues  in  depth. 

Misdesign  of  questionnaires,  often  as  a  result  of  a  lack  of 
pilot  testing  that  may  be  a  result  of  a  lack  of  sufficient  time. 

Data  gathering  survey  misdesign,  particularly  the  inappro- 
priate allocation  of  survey  resources  that  result  in  an  over- 
sampling  of  the  most  significant  groups,  as  was  the  case  with 
the  attempted  application  of  the  Watts-Conlisk  sampling  model 
to  the  New  Jersey  income  maintenance  experiment. 

Ignorance  of  or  failure  to  explore  relevant  literature  and 
data  concerning  similar  programs  or  phenomena  being  evaluated  — 
a  point  often  stressed  by  J.S.  Tukey. 

Failure  to  recognize  and  design  around  the  Hawthorne  effect 
and  other  types  of  self-fulfilling  and  self-selecting  and  de- 
selecting phenomena. 

Voluntarism  in  the  selection  of  surveyed  experimental  and 
control  groups  leading  to  biased  results,  as  when  unsuccessful 
programs  choose  not  to  be  part  of  an  evaluation. 

Premature  and  inappropriate  data  collection  efforts  of 
perishable  information  that  cannot  subsequently  be  readily  cor- 
rected, as  in  the  collection  of  significant  amounts  of  unanalyzable 
data  on  planned  variation  in  the  Follow- Through  Experiment. 

Confusion  of  a  program  inventory  for  a  program  evaluation 
—  the  substitution  of  merely  descriptive  case  studies  for  analysis 
of  program  effectiveness  and  efficiency  with  respect  to  goals  and 
resources  use. 

Confusion  of  needs  assessment  for  an  evaluation,  assuming 
that  determination  of  the  relevance  of  program  design  to  an 
established  set  of  needs  constitutes  an  adequate  evaluation. 

Confusion  of  program  outputs  for  impacts,  as  is  frequently 
the  case  in  school  evaluations  that  boast  of  increases  in  average 
daily  attendance  that  may  or  may  not  have  any  relationship  to 
desired  educational  impacts  such  as  increased  life  time  earnings. 

Contamination  of  program  results  by  formative  evaluation 
procedures  that  directly  influence  program  activities  as  part  of 
the  evaluation  processes  in  non-reproducible  ways. 
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Mis-selection  of  program  goals  and  subsequent  criteria  of 
effectiveness,  as  in  cases  where  the  program  operator's  goals 
differ  from  those  of  the  sponsors  and  possibly  those  identified 
by  the  evaluators. 

Some  of  the  specific  technical  and  managerial  approaches  which 

may  help  to  correct  these  typical  weaknesses  of  contemporary  evaluation 

research  are  described  below. 

Lack  of  Randomization 

The  excuses  most  frequently  given  for  lack  of  random  assignment 
to  experimental  and  control  groups  are  evaluation  timing,  political 
obstacles  and  resource  constraints.   (There  is  still  the  all-too-common 
unexcused  reason  that  "matching"  will  serve  the  same  purpose,  where  it 
most  often  will  not.   This  needs  to  be  corrected  by  the  further  sta- 
tistical education  of  evaluation  sponsors,  designers  and  users.) 

The  post-hoc  timing  of  many  evaluations  render  random  assignment 
subject  to  the  accessibility  of  a  "natural"  control  group  which  satisfies 
this  requirement.   Since  most  post-hoc  evaluations  are  concerned  with 
treatments  specifically  addressed  to  a  particular  "target"  population's 
problem,  it  is  unlikely  that  such  a  "natural"  control  group  that  sat- 
isfies other  experimental  conditions  can  be  readily  identified  after 
the  treatment  is  already  well  under  way.  One  policy  for  avoiding  this 
problem  is  to  require  evaluations  to  be  designed  and  executed  concur- 
rently with  the  programs  to  be  evaluated. 

Political  obstacles  to  random  assignment  exist  when  a  treatment  is 
perceived  as  very  desirable,  and  untreated  controls  perceive  themselves 
as  being  discriminated  against.  Where  "double-blind"  experimental 


325 


approaches  are  not  feasible  (as  in  most  social,  as  contrasted  to  bio- 
medical, experiments,  because  of  treatment  detectability  by  laymen), 
some  forms  of  side  payments  to  compensate  controls  for  providing 
information  while  not  receiving  treatment  have  proven  effective.   In 
operating  HUD's  Experimental  Housing  Allowance  Program's  Demand  Ex- 
periment, for  example,  Abt  Associates  Inc.  has  not  experienced  any 
significant  political  opposition  from  the  randomly  selected  controls 
not  receiving  the  allowance,  probably  because  they  receive  a  modest 
payment  (much  less  than  the  housing  allowances)  simply  to  compensate 
them  for  supplying  control  information. 

Resource  constraints  are  claimed  to  exist  where  random  selection 
of  controls  is  inconsistent  with  a  high  probability  of  obtaining  sta- 
tistically significant  quantities  of  both  experimentals  and  controls 
for  all  the  variables  of  findings  for  a  smaller  set  of  variables  versus 
le86  validity  (or  unknown  validity)  for  a  larger  set.   In  most  cases, 
sponsors  and  producers  of  evaluation  should  require  that  the  available 
surveying  resources  be  distributed  to  achieve  valid  findings  for  what- 
ever set  of  variables  can  be  accommodated,  rather  than  invalid  findings 
for  a  more  ambitious  set.  Alternatively,  exploratory  or  inventory- 
taking  evaluations  may  require  inclusion  of  all  known  variables  but  not 
generalizability  of  findings.   In  these  latter  cases,  random  assignment 
to  controls,  and  sometimes  even  controls  themselves,  may  be  sacrificed 
for  other  research  objectives,  but  the  limitations  of  the  resulting 
findings  should  be  clearly  understood  and  accepted  by  sponsors  and 
evaluators  at  the  outset. 
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Lack  of  Control  Groups 

The  absence  of  a  control  group  is  often  justified  by  lack  of 
data  gathering  resources,  political  infeasibility,  and  the  irrelevance 
of  comparisons. 

The  first  two  obstacles  have  been  discussed  above.   The  issue 
of  irrelevance  of  comparisons  requires  clarification.  When  a  pro- 
gram's relative  effectiveness  and  efficiency  are  to  be  evaluated, 
controlled  comparisons  are  being  developed,  the  experimental  method 
need  not  apply. 

Sponsors  of  evaluation  and  research  must  distinguish  among  basic 
research,  development,  test,  and  evaluation  programs.   Basic  research 
may  or  may  not  require  control  groups.  Development  and  testing  rarely 
do,  and  evaluation  usually  does.  Where  the  evaluation  aspires  to 
validity,  generalizability,  and  replicability,  control  groups  should 
be  insisted  on. 

Failure  to  Recognize  Statistical  Artifacts 

Most  of  these  kinds  of  errors  of  evaluation  can  be  avoided  by 
advanced  training  in  statistics  and  research  design.   A  straightforward 
policy  for  assuring  such  competence  is  to  require  it  of  the  technical 
director  of  all  major  evaluation  efforts.   If  this  is  impractical,  a 
somewhat  more  cumbersome  but  probably  effective  approach  would  be  to 
require  all  evaluation  research  designs  to  be  reviewed  and  accepted 
as  technically  sound  by  a  group  of  statisticians  of  proven  competence, 
institutionalized  within  some  government  agency.   OMB  clearance  of 
questionnaires  provides  both  a  good  and  a  correctable  model  for  such 
an  approach. 
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Failure  to  Select  Readily  Measurable  Measures  of  Effectiveness  Rele- 
vant to  Policy  Issues 

This  is  often  a  problem  where  evaluators  are  more  concerned  with 
contributing  to  a  discipline  or  field  of  knowledge  then  they  are  to  the 
formulation  of  a  policy  or  operation  of  a  program. 

The  most  obvious  corrective  approach  is  to  conduct  a  decision 
analysis  of  how  the  policies  or  programs  of  interest  will  actually 
be  decided  on,  before  the  evaluation  is  designed.   This  decision  anal- 
ysis should  identify  the  relevant  decision  criteria  and  supportive  in- 
formation requirements,  so  that  the  evaluation  research  can  be  specified 
to  meet  those  requirements. 

Failure  to  select  readily  measurable  measures  of  effectiveness  is 
often  the  result  of  incomplete  or  insufficiently  specific  conceptualiza- 
tion of  an  evaluation  research  problem.   For  example,  in  an  evaluation 
of  the  effectiveness  of  child  day-care  centers  in  meeting  children's 
developmental  needs,  an  incomplete  conceptualization  would  include 
only  children's  activities  in  the  centers,  and  neglect  the  parent- 
teacher-child  interactions.   Alternatively,  an  insufficiently  specific 
conceptualization  would  call  for  measuring  "all  developmental  activities' 
whatever  that  might  mean.   Priorities  of  data  collection  must  be  set  to 
make  sense  with  respect  to  a  comprehensive  conceptual  model  of  the 
phenomena  being  evaluated,  and  then  expressed  in  operationally  definable 
terms. 

Too  Broad  a  Focus 

This  is  generally  a  problem  of  either  over-ambitiousness  or  under- 
appreciation  of  the  costs  of  collecting  and  analyzing  information 
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responding  to  specific  questions,  or  both.   It  can  be  corrected,  like 
most  misallocations  of  effort,  by  setting  clear  priorities  and  esti- 
mating opportunity  costs  for  specific  tasks,  maximizing  the  accomplish- 
ment of  highest  priority  high  productivity  tasks.   Sponsors  of  evalua- 
tion sometimes  contribute  to  the  native  intellectual  imperialism  of 
some  evaluators  by  urging  the  evaluators  to  find  out  "a  lot  about  a 
lot  of  things",  with  the  likely  result  that  little  that  is  new  or 
useful  is  found  out  about  even  a  few  things.  This  process  is  en- 
couraged by  competitive  procurement,  wherein  bidders  attempt  to  outdo 
each  other  in  the  technical  accomplishments  they  promise.  Within  all 
too  rough  limits  of  credulity  there  are  apparent  rewards  for  promising 
too  broad  an  effort  at  some  presumably  technically  competent  depth, 
and  few  disincentives  to  promising  too  much. 

Organizationally,  much  could  be  done  to  reduce  these  incentives 
to  over-promises  of  excessive  breadth  by :  (1)  sponsors  specifying 
the  required  scope  or  breadth  and  faulting  bidders  for  going  far  be- 
yond it;  (2)   sponsors  evaluating  bids  on  the  basis  of  the  percentage 
of  promised  achievements  actually  realized  by  the  bidder  in  completed 
researches;  and  (3)  continuous  and  candid  communication  between  spon- 
sors and  researchers  throughout  the  period  of  contract  performance  con- 
cerning research  priorities  and  the  cost  and  tradeoffs  involved  in 
realizing  them. 

Note  that  a  pro-forma  monthly  progress  report  does  not  often 
achieve  the  third  objective.  Face-to-face  meetings  between  sponsor  and 
researcher  are  needed  to  develop  enough  mutual  trust  and  confidence  for 
the  evaluator  to  be  able  to  risk  admitting  that  the  goals  of  the 
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evaluation  had  shifted  and  with  the  task  priorities,  without  the  re- 
searchers resorting  to  legalism  or  unresponsiveness. 

Too  Narrow  a  Focus 

Partly  in  response  to  having  been  "burned"  by  the  non-results  of 
an  evaluation  too  broadly  focused,  or  because  they  think  they  know  what 
the  key  variables  are,  sponsors  and  evaluators  may  insist  on  too  narrow 
a  focus  and  prematurely  exclude  some  data  and  variables  that  might  have 
yielded  significantly  different  results.  Truncated  samples,  arbitary 
and  unexplained  elimination  of  outliers,  and  ignoring  of  non-responding 
data  sources  are  all  examples. 

Ideally,  given  sufficient  time  and  resources,  all  evaluation  re- 
search should  be  executed  in  at  least  three  cycles  —  the  first  broad 
and  shallow  to  maximize  the  probability  of  capturing  the  critical 
variables,  the  second  more  narrow  and  with  an  in-depth  analysis  of 
the  hypotheses  generated  in  the  first  cycle,  and  then  a  number  of 
almo6t-as-deep  parallel  analyses  to  check  external  validity  and  reli- 
ability. 

Since  this  kind  of  iterative  process  is  not  always  affordable, 
it  may  be  helpful  to  attempt  to  compress  the  cycles  to  within  a  given 
evaluation,  or  to  use  analogical  other  efforts  to  simulate  the  first 
and  third  cycles.   In  any  case,  a  thorough  and  broad  literature  search 
should  reduce  (but  will  not  eliminate)  the  probability  that  a  too  nar- 
row focus  has  excluded  potentially  significant  factors. 

Too  Short  Duration 

This  is  sometimes  the  sponsor's  fault  in  insisting  on  a  one-year 
duration  for  contracting  or  budgeting  convenience,  and  sometimes  the 
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researchers'  fault  when  through  poor  planning  much  of  the  contract  time 
is  wasted  and  the  long  lead  time  tasks  must  be  compressed  to  a  duration 
inimical  to  their  successful  completion. 

What  is  needed  here  is  some  standardization  of  the  lead  times  of 
the  lengthier  evaluation  research  tasks,  such  as  field  testing  of  ques- 
tionnaires, data  reduction  and  cleaning,  etc.   Further,  there  needs  to 
be  a  better  appreciation  of  both  the  advantages  and  the  elapsed  time 
requirements  of  longitudinal  research  —  classical  "before  and  after" 
treatment  evaluations  in  school  settings  are  impractical  within  one 
year,  since  it  is  unlikely  that  this  duration  permits  the  necessary 
planning  and  design  before  baseline  data  gathering  or  the  necessary 
data  reduction  and  analysis  time  after  the  after-treatment  data  is 
collected. 

Misdesign  of  Questionnaires 

This  or  other  data- gathering  instruments  can  be  corrected  by  pilot 
testing,  provided  there  is  sufficient  time  available.   Much  of  the  pilot 
testing  time  might  be  saved  if  standards  of  competence  in  instrument 
design  were  more  strongly  insisted  upon  by  sponsors,  since  almost  every- 
one believes  that  they  can  design  an  effective  questionnaire. 

There  is  remarkably  little  questionnaire  research  built  into  most 
evaluations,  yet  the  results  of  the  entire  effort  are  usually  criti- 
cally dependent  on  the  quality  of  these  instruments.   This  is  an  ac- 
tivity in  which  there  appears  to  be  very  little  technique  transfer 
between  the  masterful  practitioners  and  the  neophytes  —  again,  because 
neophytes  believe  it  is  easy  (and  it  ±s^  easy  to  do  it  badly) .   Standard- 
ization of  competent  instrument  design  approaches  and  their  dissemination 
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might  help.   Sponsors  could  raise  standards  by  requiring  evaluation 
researchers  to  submit  examples  of  previous  instrument  designs,  to- 
gether with  critiques  of  their  strengths  and  weaknesses  in  the  light 
of  actual  use. 

Data  Gathering  Survey  Misdesign 

Here  again,  many  marginal  evaluators  fail  to  be  aware  of  or  exploit 
the  great  mass  of  survey  research  and  sampling  methodology  is  quite 
adequate  for  meeting  the  technical  demands  of  most  evaluation  researchers. 
The  major  problem  is  that  the  full  competence  of  this  sub-field  of  social 
science  is  rarely  fully  applied. 

The  state  of  the  art  under-utilization  problem  can  be  addressed  by 
sponsors  imposing  higher  standards  of  survey  competence  on  the  proposing 
evaluation  researchers.  Appropriate  course  work  and  experience  in  sta- 
tistics, sampling,  design,  and  instrumentation  should  be  specified  for 
survey  task  leaders. 

For  the  more  sophisticated  survey  design  problems,  the  uncer- 
tainty concerning  the  optimal  choices  of  defining  factors  may  be  par- 
tially overcome  by  pilot  testing  and/or  simulation. 

Ignorance  of  the  Relevant  Literature 

This  is  correctable  by  effort,  insistence  on  it  and  demonstration 
of  its  results. 

Failure  to  Recognize  and  Design  Around  the  Hawthorne  Effect 

Again,  this  is  a  matter  of  education  of  the  producers  and  consumers 
of  evaluation  research,  and  the  enforcement  of  competency  standards 
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reflecting  the  best  that  is  known  concerning  research  design.  Campbell 
and  Stanley  61/  offer  numerous  methods,  such  as  the  Solomon  four-group 
design,  for  dealing  with  this  similar  testing  artifacts. 

Voluntarism  in  Selection 

This  is  often  as  much  a  policy  as  it  is  a  technical  issue.   Spon- 
sors wishing  to  avoid  offending  program  operators  may  back  off  from 
insisting  on  evaluation  of  programs  or  sites  that  resist  it.   Sponsors 
may  intentionally  or  inadvertently  select  sites,  cases,  or  projects  in 
biased  ways  influencing  the  results  of  evaluation. 


61/  Campbell,  D.T.  and  Stanley,  J.C.   Experimental  and  quasi- 
experimental  designs  for  research  on  teaching.   In  N.L.  Gage,  ed. 
Handbook  of  research  on  teaching.  Chicago,  Rand  McNally,  1963. 
(Reprinted  as  Experimental  and  quasi-experimental  design  of  re- 
search.  Chicago,  Illinois,  Rand  McNally,  1966.) 
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"EVALUATING  EDUCATION  PROGRAMS ARE  WE  GETTING  ANYWHERE? 

JOHN  W.  EVANS,  OFFICE  OF  EDUCATION  * 


From  the  Educational  Researcher,  published  by  the  American 
Educational  Research  Association.   Remarks  delivered  were 
based,  in  part,  on  this  paper. 
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Evaluating  Education  Programs 
Are  We  Getting  Anywhere?  ' 


JOHN  W.  EVANS 
U.S.  Office  of  Education 


Now  seems  to  be  a  time  when 
basic  reassessments  are  in 
order;  so  it  is  only  appropriate  for 
those  of  us  concerned  with  educa- 
tional evaluation  to  take  stock  of  our 
own  endeavors  and  ask  whether  we 
are  really  getting  anywhere  in  our 
efforts  to  assess  the  effectiveness  of 
educational  programs,  or  whether 
all  the  current  talk  and  frenetic  ac- 
tivity is  much  ado  about  nothing. 
The  answer  to  this  question  is  by  no 
means  obvious,  even  though  it  is  the 
kind  of  rhetorical  question  that  pa- 
pers like  this  typically  answer  with 
rosy  if  vague  and  overqualified  bro- 
mides. I'm  going  to  hedge  and 
qualify  too,  but  my  basic  conclusion 
is  that  we  are  getting  somewhere. 
I  believe  important  progress  has 
been  made  in  recent  years  in  edu- 
cational evaluation  in  a  variety  of 
ways  which  I  intend  to  specify,  but 
the  educational  evaluation  scene  is 
not  an  untroubled  one.  Far  from  it: 
there  are  serious  new  problems  that 
threaten  the  progress  that  has  been 
made,  and  I  intend  to  talk  about 
those  also. 


A  Brief  Look  at  History 

We  can  begin  with  some  simple 
and  chastening  assertions.  First,  the 
history  of  significant  evaluations  of 
educational  programs  is  brief  and 
thin.  In  this  respect  educational 
evaluation  is  of  a  piece  with  the 
evaluation  of  social  action  programs 
generally.  If  we  look  back  over  the 
history  of  Federal  efforts  in  the  so- 
cial program  area,  even  back  into 
the  period  of  the  New  Deal  and  on 
up  through  the  Great  Society  pro- 
grams of  the  sixties,  we  are  forced 
to  acknowledge  that  virtually  all  the 


original  decisions  by  the  Congress 
and  the  Executive  Branch  of  the 
Federal  government  to  initiate  pro- 
grams in  the  areas  of  education, 
manpower  and  poverty,  and  the 
later  decisions  to  continue,  expand, 
or  terminate  these  programs,  were 
taken  with  scarcely  any  knowledge 
of  the  size,  character,  and  location 
of  the  problems,  or  the  likely  effec- 
tiveness of  proposed  programs  to 
remedy  them.  Once  instituted,  such 
programs  were  only  rarely  subjected 
to     rigorous    objective    evaluation. 

Second,  failure  to  evaluate  edu- 
cation programs  is  a  shortcoming 
not  limited  to  the  Federal  Govern- 
ment. States  and  localities,  which 
supply  95%  of  the  funds  for  public 
education,  have  done  little  to  evalu- 
ate the  effectiveness  of  their  school 
systems  and  educational  approaches. 

Third,  with  a  few  notable  excep- 
tions, academic  social  scientists, 
traditionally  accustomed  to  the  re- 
search style  of  the  individual  schol- 
arly grant,  have  been  largely  pre- 
occupied with  «Jisciplinary  issues 
and  basic  research,  rather  than 
"with  actual  evaluations  of  ongoing 
educational  programs. 

Such  have  been  our  efforts  to  for- 
mally evaluate  most  of  our  national 
education  and  other  domestic  pro- 
grams. Yet,  in  less  than  ten  years 
we  have  gone  from  a  dearth  of  eval- 
uation activity  to  a  situation  where 
evaluation  is  now  all  the  rage.  Even 
though  the  number  of  professional 
association  meetings  currently  being 
devpted  to  evaluation  are  exag- 
gerated indicators  of  the  amount  of 
useful,  policy  relevant  evaluation 
now  going  on,  there  is  no  doubt  that 
the  change  has  been  real  and  sub- 
stantial. 


What  has  accounted  for  this  rela- 
tively sudden  upsurge,  and  does  it 
amount  to  real  progress— progress 
in  conducting  sound  evaluations  and 
making  them  pajt  of  the  policy 
process?  There  is  no  simple  answer 
to  this  question,  but  from  my  vant- 
age point  there  seem  to  be  several 
important  factors  which  have  ac- 
counted for  the  increased  concern. 

First,  there  is  the  long-run,  cumu- 
lative effect  of  the  presence  and 
force  of  social  science  in  our  society. 
Social  scientists  have  long  been 
chiding  administrators,  policy 
makers,  and  Congressmen  to  rely 
less  on  subjective  and  political  rea- 
sons for  making  decisions,  allo- 
cating resources,  and  developing 
programs,  and  instead  to  make 
more  use  of  the  research  methods 
and  findings  of  social  science.  These 
entreaties  have  often  turned.out  to 
be  more  rhetorical  than  substantive 
once  the  challenge  was  taken  up, 
But  they  have  not  gone  without  ef- 
fect on  the  policymakers,  who  have 
been  made  to  feel  increasingly 
guilty  about  not  formulating  policy 
and  making  decisions  in  a  more 
rational  way. 

Second,  there  has  been  a  gradual 
transformation  in  the  intellectual 
make-up  of  the  kind  of  people  who 
have  found  themselves  in  both  ap- 
pointed and  elected  offices.  While  it 
remains  true  that  raw,  unreasoned, 
political  interest  still  is  the  main 
factor  in  many  decisions  made  by 
both  the  Executive  and  Legislative 
branches  of  the  Federal  Govern- 
ment, it  is  also  true  that  the  last  10 
to  20  years  have  seen  a  significant 
increase  in  the  number  of  people  in 
such  positions  who  want  to  attack  a 
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problem  by  asking  about  the  scope 
of  its  dimensions  and  the  effective- 
•  ailable  methods  of  treat- 
ing it  — that  is.  people  who  want  to 
try  to  rationalize  the  policy  process. 

Certainly  another  important  fac- 
tor in  accounting  for  the  upsurge  of 
concern  with  evaluation  at  the  Fed- 
eral le\el  was  the  implementation  of 
the  Program  Planning  and  Budget- 
ing System  (PPBS).  This  approach 
to  analyzing  and  making  decisions 
about  program  budget  le\els  is  a 
radical  departure  from  the  tradi- 
tional incremental  approach.  It 
turns  the  focus  away  from  the  stand- 
ard administrative  budget  categories 
toward  the  objectives,  methods,  and 
outcomes  of  programs.  Such  a  shift 
automatically  brought  the  need  for 
data  on  program  effectiveness  to  the 
fore. 

Finally,  there  is  the  accountability 
movement  in  education.  It  is  hard  to 
know  whether  this  movement  is  a 
cause,  an  effect,  or  merely  an  indi- 
cator of  the  increased  interest  in 
evaluation  In  any  case,  once  pres- 
ent, it  has  become  a  force  in  its  own 
right 

But  what  have  all  these  changes 
in  analytical  approach  and  ex- 
pressed concern  amounted  to  be- 
yond heating  up  the  atmosphere  and 
expanding  the  rhetoric?  What  have 
they  resulted  in  that  allows  one  to 
conclude  that  some  actual  progress 
in  educational  evaluation  is  being 
made?  To  answer  these  questions 
let  us  look  at  three  aspects  of  the 
evaluation  process:  first,  the  in-puts 
and  resources:  second,  the  meth- 
odology: and  third,  actual  evalua- 
tion studies  and  their  results. 

Some  Indicators  of  Progress 
Resources  for  Evaluation 
and  the  Avenues  of  Impact 

There  have  been  major  increases 
in  the  wherewithal  required  for  eval- 
uations to  get  done,  and  important 
improvements  in  the  organizational 
location  of  the  evaluation  function 
in  the  Federal  Government's  deci- 
sion-making apparatus.  As  the  larg- 
er social  changes  I  noted  earlier 
have  heightened  concern  with  eval- 
uation generally,  the  Congress  has 
tired  of  listening  to  requests  for  in- 
creased appropriations  based  on 
anecdotes  and  testimonials,  and  has 
increasingly   demanded  that   Execu- 


tive branch  agencies  produce  data 
on  the  effectiveness  of  their  pro- 
grams. For  its  part,  the  Congress 
has  substantially  increased  funds 
and  personnel  to  the  domestic  agen- 
cies for  evaluation.  In  1965.  the 
Departments  of  Labor  and  HEW 
had  available  less  than  S5.0  million 
for  program  evaluation.  By  I9"4  this 
figure  had  increased  more  than  ten- 
fold to  more  than  $50.0  million. 

There  have  been  equally  impor- 
tant changes  in  the  organizational^ 
location  of  the  evaluation  function. 
As  those  of  us  who  have  worked  in 
program  agencies  can  testify,  one  of 
the  indispensable  prerequisites  if 
evaluation  is  to  impact  on  decisions 
is  that  it  must  be  an  integral  part  of 
top  management's  decision-making 
structure.  Yet,  evaluation,  like  re- 
search, has  often  been  buried  in  the 
bowels  of  program  agencies  and  not 
only  has  received  the  leftovers  in 
fiscal  and  personnel  resources  but 
also  has  had  little  opportunity  to 
make  a  meaningful  input  into  the 
policy  process.  Even  this  is  now 
changing.  Most  of  the  major  Fed- 
eral agencies  now  have  an  Assistant 
Secretary  for  Planning  and  Evalua- 
tion or  its  equivalent,  and  the  shoe 
is  now  on  the  other  foot.  Instead  of 
having  to  plead  for  more  money  and 
people  and  the  chance  to  participate 
in  the  decision-making  process.  e\  al- 
uation  specialists  are  now  under 
pressure  to  produce  and  to  justify 
their  claims  of  utility  and  relevance 
As  mans  of  us  have  found  out,  de- 
manding our  place  in  the  sun  is  a  lot 
easier  than  justifying  it. 

Perhaps  the  most  important  thing 
to  come  out  of  all  these  resource 
increases  and  organizational  changes 
"is  that  the  basic  dialogue  of  man- 
agement has  begun  to  change  from 
considerations  of  how  big  a  pro- 
gram's budget  should  be  and  of  the 
constituency  pressures  for  its  con- 
tinuation, to  considerations  of  ob- 
jective evidence  of  performance  and 
indicators  of  program  effectiveness. 

The  Lse  of  More  Sophisticated 
Methodology  in  Evaluation 

1  want  to  touch  briefly  on  two 
important  methodological  advances. 
The  first  is  the  appearance  of  efforts 
in  large  scale  national  evaluations 
to  use  the  classic  model  of  experi- 
mental    design     with     randomized 


treatment  and  control  groups. 

Since  virtually  all  education  pro- 
grams have  their  committed  advo- 
cates and  strong  detractors,  and  are 
in  this  sense  inherently  controver- 
sial, it  is  inevitable  that  evaluations 
of  them  will  also  be  controversial. 
Any  evaluation  that  finds  a  pro- 
gram successful  "will  be  attacked 
by  the  program's  detractors,  and 
any  evaluation  that  finds  the  pro- 
gram unsuccessful  will  be  denounced 
b>  its  advo'cates.  As  Rossi  (1972) 
■has  put  it,  "No  good  evaluation 
goes  unpunished."  In  such  cases, 
the  controversy  will  not  be  waged 
directly  over  what  is  truly  at  stake, 
namely  the  disagreeableness  of  the 
findings,  but  instead  will  take  the 
form  of  an  attack  on  the  validity  of 
the  evaluation's  methodology  Acri- 
monious debates  will  rage  through 
the  daily  press  and  the  professional 
journals  over  sample  size  in- 
adequacies, non-representativeness, 
culture-biased  measurement  instru- 
ments, failures  to  meet  the  assump- 
tions of  parametric  statistical  mod- 
els, and  the  like.  Since  no  evaluation 
can  be  flawless,  especially  those  car- 
ried out  in  the  real  world  of  class- 
rooms and  communities,  evaluation 
specialists  will  never  escape  these 
post-evaluation  debates  — nor.  in- 
deed, should  they.  But  to  increase 
the  justification  for  using  formal 
empirical  evaluations  as  a  basis  for 
policy  decisions,  it  is  important  that 
each  evaluation  be  as  methodologi- 
cally strong  as  possible. 

The  major  weakness  in  most  eval- 
uation designs  relates  to  the  use  of 
control  groups.  The  feature  of  edu- 
cation evaluations  that  has  proven 
most  vulnerable  to  both  well-moti- 
vated and  not-so-well-motivated 
attack  is  the  comparability  between 
treatment  and  control  groups.  Once 
evaluations  move  beyond  the  primi- 
tive efforts  to  conduct  site  visits  or 
simply  collect  data  on  a  before- 
after  basis,  the  evaluator  and  the 
design  he  employs  must  confront  the 
fundamental  problem  of  providing 
some  estimate  of  what  would  have 
happened  in  the  absence  of  the  pro- 
gram he  is  evaluating.  There  are.  of 
course,  a  variety  of  ways  to  deal 
with  this  problem,  which  include 
comparison  with  national  norms, 
comparison  with  previous  years' 
scores,  the  use  of  a  matched  com- 


336 


parison  group,  etc.  But  the  effects  of 
education  programs  are  seldom  dra- 
matic, and  the  small  differences 
they  are  likely  to  make  can  easily  be 
either  overestimated  or  missed  en- 
tirely by  comparing  the  treatment 
group  with  a  non-comparable  group. 
The  only  truly  satisfactory  way  of 
dealing  with  this  problem,  of  course, 
is  through  randomly  assigned  treat- 
ment and  control  groups.  Campbell 
(1969)  has  written  eloquently  and 
extensively  on  this  issue,  urging  that 
educational  and  other  kinds  of  pro- 
grams be  structured  at  the  outset  to 
allow  this  kind  of  evaluation  design. 
He  and  I  have  debated  the  question 
of  how  acceptable  evaluations  are 
which  fall  short  of  this  standard 
(Campbell  &  Erlebacher;  Evans  & 
Schiller,  1970).  My  view  is  that 
there  will  be  many  instances  where 
this  obviously  stronger  design  is  not 
feasible,  and  that  rather  than  throw 
up  our  hands  and  withdraw  from 
the  arena  because  we  cannot  have 
random  assignment,  we  must  carry 
out  whatever  kind  of  evaluation  is 
feasible  and  useful  within  the  time 
constraints  of  the  policy  process  and 
make  the  best  use  of  it  we  can.  It  is 
my  experience  that  even  fairly  prim- 
itive designs  are  likely  to  provide 
better  data  for  decision-making 
than  the  subjective  impressions  and 
partisan  arguments  normally  used. 
My  point  here  is  simply  that  while 
we  must  be  willing  to  do  what  we 
can  to  improve  the  decision-making 
process,  I  nevertheless  agree  with 
Campbell  that  we  should  certainly 
strive  to  use  the  superior  method  of 
randomized  experimental  design 
whenever  we  can.  And  I  would  re- 
port that  the  Evaluation  Unit  at  the 
Office  of  Education  has  been  able  to 
mount  two  such  randomized  design 
evaluations  of  the  Emergency  School 
Assistance  Program  in  the  last  two 
years.  One  is  completed  and  the 
other  is  still  in  process.  In  a  forth- 
coming paper  in  which  "he  analyzes 
efforts  to  introduce  and  assess  in- 
novations across  a  broad  variety  of 
fields  from  medicine  to  education, 
Mosteller  notes  that  this  is  appar- 
ently the  first  time  in  education  in 
which  a  major  evaluation  was  car- 
ried out  using  this  type  of  design. 

The  other  methodological  ad- 
vance I  want  to  comment  on  is  the 
emergence  of  experimentation  as  a 


precursor  to  the  full  fledged  intro- 
duction of  major  new  programs. 
This  could  be  one  of  the  most  im- 
portant developments  in  our  time  if 
it  takes  hold  and  is  actually  used  — 
and  those  are  two  big  ifs. 

To  understand  what  social  or  edu- 
cational experimentation  entails  and 
how  it  may  be  of  great  value  in 
developing  educational  policies  and 
programs,  it  may  be  helpful  to  look 
at  what  has  happened  in  the  field  .of 
compensatory  education.  In  the  ear^  . 
ly  sixties  the  country  belatedly  rec- 
ognized the  existence  of  the  dis- 
advantaged child.  It  was  acknowl- 
edged that  about  a  fifth  of  our  chil- 
dren arrive  at  the  first  grade  with 
educational  deficits  that  are  meas- 
urable even  at  that  time;  that  as 
they  progress  through  school  the 
achievement  gap  between  them  and 
their  middle  class  peers  widens;  that 
a  great  many  do  not  learn  to  read, 
write,  or  calculate  adequately;  that 
they  drop  out  of  school  in  large 
numbers;  and  that  as  a  consequence 
of  these  deficits  in  basic  skills  and 
credentials  they  are  unable  to  pur- 
sue postsecondary  education  or  to 
form  either  a  lasting  or  satisfying 
attachment  to  the  labor  market. 
What  is  worse,  these  educational 
disadvantages  are  passed  on  cul- 
turally to  their  children,  and  a  sig- 
nificant portion  of  the  society's  pop- 
ulation is  caught  up  in  a  cycle  of 
intergenerational  poverty. 

Once  these  problems  were  recog- 
nized, the  sense  of  political  urgency 
was  irresistable,  and  the  country 
rushed  to  pass  major  legislation  and 
initiate,  among  other  things,  major 
early  childhood  compensatory  edu- 
cation programs.  In  this  atmo- 
sphere of  rushing  to  solve  the  prob- 
lem, only  the  faintest  of  voices  were 
heard  asking  the  unaskable  ques- 
tions: Did  we  really  know  what  we 
were  doing?  Did  we  really  have  ef- 
fective program  models  and  tech- 
niques which  could  remediate  the 
educational  deficits  of  these  chil- 
dren? 

The  evaluations  that  have  since 
been  carried  out  on  these  national 
programs,  principally  Head  Start 
and  Title  I  (e.g.,  Westinghouse, 
1969;  and  Wargo,  1972),  indicate 
that  we  did  not  know  what  we  were 
doing.  While  our  ideology  was  laud- 
able and  our  motivations  pure,  our 


programmatic      know-how      was 
skimpy. 

The  problem  is  that  once  large 
national  programs  are  put  into 
place,  the  political  force  of  their 
authorship  and  the  pressure  from 
their  constituencies  for  continued 
funding  make  it  almost  impossible 
to  even  acknowledge  publicly  that 
they  may  not  be  effective,  much  less 
attempt  to  alter  them  in  some  fun- 
damental, way  to  make  them  so. 
This  is  where  a  strategy  of  experi- 
mentation can  be  of  great  value. 
After  there  is  broad  recognition  of 
a  major  educational  or  social  prob- 
lem, if  we  can  force  ourselves  to 
recognize  that  we  may  not  know 
how  to  solve  it,  and  if  instead  of 
going  directly  to  a  massive  national 
program,  we  wiy  initiate  a  con- 
trolled experiment  in  which  we  de- 
velop and  test  the  relative  effective- 
ness of  alternative  programmatic 
techniques,  we  can  reap  a  number  of 
benefits.  First,  if  the  results  of  that 
experiment  show  that  we  have  not 
yet  achieved  effective  program 
models  and  techniques,  it  is  politi- 
cally possible  to  admit  this  and  go 
back  to  the  drawing  board  to  devel- 
op them.  Moreover,  we  can  take 
considerable  satisfaction  in  the 
knowledge  that  we  are  not  com- 
mitted to  the  continued  expenditure 
of  large  resources  on  efforts  we 
know  to  be  ineffective.  On  the 
other  hand,  if  the  experiment  is  suc- 
cessful, we  can  go  forward  with  a 
large  national  service  program  rea- 
sonably confident  that  the  massive 
resources  we  will  be  devoting  to  the 
problem  will  have  a  good  chance  of 
actually  making  a  dent  in  it. 

The  logic  of  using  experimenta- 
tion as  a  precursor  to  national 
program  implementation  may  seem 
compelling  enough,  but  the  rush  to 
adopt  this  strategy  has  not  exactly 
been  a  stampede.  Analyzing  this 
situation,  Timpane  (1970)  has  made 
a  series  of  observations  on  the  po- 
tential of  social  experimentation 
which  I  have  jokingly  called  "Tim- 
pane's  law."  His  conclusion  is  that 
if  there  is  enough  interest  in  some 
problem  to  support  a  major  social 
experiment,  then  the  interest  will  be 
so  great  that  no  one  will  be  willing 
to  wait  for  the  conclusion  of  the  ex- 
periment  before  passing  legislation 
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to  implement  a  national  program. 
On  the  other  hand,  if  there  is  not 
broad  concern  over  the  problem, 
then  there  won't  be  enough  interest 
in  Congress  to  support  the  funding 
of  an  experiment  on  it.  Either  way 
there  is  no  experiment. 

There  is  truth  as  well  as  humor  in 
Timpane's  law,  but  some  significant 
efforts  at  experimentation  in  educa- 
tion are  nevertheless  occurring,  and 
we  should  not  overlook  their  impor- 
tance both  as  early  prototypes  of 
what  could  become  a  fundamentally 
new  way  of  approaching  the  devel- 
opment and  initiation  of  education 
programs,  and  for  the  contribution 
they  have  made  to  the  particular 
educational  issues  they  address.  I 
would  cite  two  such  efforts.  The  first 
is  the  Follow  Through  program  and 
the  second  is  the  OEO  experiment 
on  Performance  Contracting. 

Follow  Through  was  originally  in- 
tended to  be  a  follow-up  service 
program  to  reinforce  whatever  gains 
were  made  in  Head  Start;  but  by 
the  time  its  first  appropriation  in 
1968  had  passed  through  the  vari- 
ous budget  cutting  phases,  the  initial 
request  of  $120  million  had  been 
reduced  to  $15  million.  Realizing 
that  it  made  no  sense  to  mount  a 
service  program  that  would  address 
only  one  percent  of  the  target  popu- 
lation, the  program  staff  shifted  the 
focus  away  from  service  delivery  to 
the  development  and  evaluation  of 
alternative  compensatory  education 
models.  This  effort  at  educational 
experimentation  has  been  plagued 
by  staff  shortages,  administrative 
difficulties,  and  continued  unclarity 
over  what  Follow  Through's  true 
mission  is  and  how  it  should  be  car-  . 
ried  out.  Nonetheless,  despite  these 
problems,  some  evaluation  findings 
are  now  beginning  to  emerge  that 
are  precisely  the  kinds  of  outcomes 
we  would  expect  from  a  planned 
variation  experiment.  Some  of  the 
program  models  are  showing  the 
ability  to  produce  cognitive  and  af- 
fective gains  that  are  larger  than 
those  we  have  seen  in  most  com- 
pensatory education  programs.  Oth- 
er models  are  producing  gains  that 
are  just  about  what  one  would  ex- 
pect from  the  normal  school  expe- 
rience, while  still  others  are  appar- 
ently so  ineffective  that  the  children 
in  the  control  group  are  educational- 


ly belter  off  than  those  in  the  model 
programs.  If  these  findings  hold  up 
in  the  subsequent  waves  of  the  long- 
itudinal evaluation,  we  should  have 
a  much  better  basis  on  which  to  pro- 
ceed programmatically  in  the  area 
of  early  childhood  compensatory 
education. 

The  OE  experiment  on  Perform- 
ance Contracting  grew  out  of  the 
kind  of  situation  which  should  call 
for  an  experiment".  About  fourvyears 
ago  a  number  of  educational  tech- 
nology firms  were  promoting  the  ■ 
ability  of  their  techniques  to  pro- 
duce large  gains  in  reading  and 
math  among  disadvantaged  chil- 
dren. Interest  in  performance  con- 
tracts began  to  sweep  through  pub- 
lic school  systems  with  large  popu- 
lations of  educationally  disadvan- 
taged children.  The  attraction  to 
performance  contracting  w;is  based 
on  a  number  of  factors.  It  was  at 
this  time  that  the  disillusionment 
about  public  education  that  flowed 
from  the  Coleman  and  later  analy- 
ses was  approaching  its  peak.  The 
siren  of  performance  contracting 
was  especially  seductive  at  this 
time  because  it  said:  "Not  only  is  it 
possible  to  remediate  the  deficits  of 
disadvantaged  children,  but  we  have 
the  techniques  to  do  it,  we  are  ready 
to  come  into  your  schools  and  im- 
plement it,  it  is  no  more  expensive 
than  your  present  per  pupil  expendi- 
ture, and  we  will  sign  a  binding  con- 
tract with  you  which  says  that  if  we 
don't  produce  significant,  independ- 
ently measured  gains  in  reading  and 
math,  you  don't  have  to  pay  us." 
Small  wonder  that  these  blandish- 
ments triggered  a  rush  to  the  per- 
formance contractor's  door. 

But  there  were  also  strident  crit- 
ics of  performance  contracting, 
mainly  the  teachers'  unions,  who 
argued  that  performance  contract- 
ing was  an  illusory  panacea  and 
that  it  would  dehumanize  the  learn- 
ing process.  Depending  on  who  won 
the  argument  — that  is,  who  shouted 
the  loudest— it  seemed  that  per- 
formance contracting  was  destined 
to  be  either  prematurely  buried  or 
unjustifiably  expanded  into  a  na- 
tional movement. 

Noting  that  these  unfounded 
claims  and  counter  charges  were 
precisely  the  circumstances  that  cull 
for   an   experiment,    the   evaluation 


staff  at  OEO  designed  and  carried 
out  such  an  effort,  underwriting  and 
independently  evaluating  seven  dif- 
ferent performance  contracting 
firms.  The  results  of  the  evaluation 
showed  that  none  of  the  perform- 
ance contract  models  was  able  to 
produce  reading  and  math  gains  sig- 
nificantly better  than  the  results 
achieved  through  the  regular  public 
school  methods.  It  is  hard  to  predict 
what  trie,  outcome  of  the  debate 
would  have  been  had  the  experi- 
ment not  been  done. 

Studies  and  Results 

Given  that  educational  evaluation 
has  shown  progress  in  the  funding 
support  it  has  attracted,  in  the  sheer 
amount  of  evaluation  activity  that 
is  going  on,  in^  the  organizational 
position  that  evaluators  hold  in  gov- 
ernment agencies,  in  the  demand  for 
evaluation  results  by  the  Congress, 
and  in  the  use  of  more  sophisticated 
designs  in  the  Conduct  of  evalua- 
tions, what  has  actually  been  done 
by  way  of  major  evaluation  studies 
that  have  important  policy  implica- 
tions? It  is  the  completion  of  such 
actual  studies,  after  all,  that  is  the 
outcome  measure  for  evaluating 
evaluation. 

First,  there  is  the  Coleman  Re- 
port itself  which,  while  it  does  not 
evaluate  a  specific  education  pro- 
gram, nevertheless  is  fundamentally 
an  evaluative  analysis  assessing  the 
effects  of  what  had  traditionally 
been  regarded  as  some  of:the  most 
important  independent  variables  in 
the  educational  process.  Notwith- 
standing the  continuing  debates 
over  its  methodological  shortcom- 
ings, few  would  deny  that  it  is  a 
landmark  study  which  has  caused 
educational  theorists  to  reassess 
their  fundamental  beliefs  and  strat- 
egies, and  legislators  to  reexamine 
their  unquestioned  faith  in  educa- 
tional programs  and  appropriations. 

Second,  the  OEO  evaluation  of 
Head  Start,  usually  referred  to  as 
the  Westinghouse  Report,  which 
was  also  the  subject  of  intensive 
methodological  scrutiny  and  debate, 
is  one  of  a  number  of  studies  of  ear- 
ly childhood  compensatory  educa- 
tion programs  which  shook  us  out 
of  our  complacent  belief  that  popu- 
lar, well  motivated  programs  are 
necessarily  effective  in  remediating 


338 


the    educational    deficits    of    poor 
kids. 

Third,  an  evaluation  of  the  Emer- 
gency School  Assistance  Program 
found  that  this  moderately  funded 
and  locally  generated  collection  of 
projects  was  able  to  significantly 
increase  the  achievement  levels  of 
black  male  teenagers,  and  thus,  by 
indicating  that  compensatory  educa- 
tion in  the  public  schools  is  possible. 
was  a  welcome  contradiction  to  the 
largely  negative  findings  of  so  many 
of  the  earlier  studies. 

Fourth,  and  in  the  same  vein,  an 
early  evaluation  of  the  Upward 
Bound  program  found  that  this  pro- 
gram was  effective  in  persuading 
low  income  high  school  youngsters 
to  attend  college,  in  keeping  them 
there,  and  in  graduating  them  at  a 
rate  which  made  the  program  cost- 
beneficial. 

I  have  already  mentioned  the 
major  evaluations  of  the  Follow 
Through  Program  and  Performance 
Contracting.  I  don't  wish  to  extend 
this  list  indefinitely,  mainly  because 
I  couldn't  if  I  wanted  to.  But. I  do 
want  to  make  the  point  thai  if  we 
ask  whether  all  the  hoopla  of  edu- 
cational evaluation  has  amounted  to 
anything  more  than  increases  in 
funds,  data  gathering,  and  profes- 
sional meetings,  the  answer  is  "yes  " 
There  is  far  less  on  the  production 
ledger  of  educational  evaluation 
than  there  should  be.  but  indica- 
tions of  important  progress  are  by 
no  means  lacking. 
A  Look  at  the  Future: 
Prospects  and  Problems 

This  recitation  of  progress  makes 
things  sound  much  better  than  they 
are.  To  be  sure,  the  progress  is  real; 
but  in  the  last  few  years  a  number 
of  new  problems  have  arisen  that 
must  be  solved  if  the  use  of  evalua- 
tion in  the  policy  process  is  to  prog- 
ress beyond  its  present  promising 
but  inchoate  state.  I  call  them  new 
problems  because  it  is  important  to 
distinguish  them  from  the  kind  of 
problems  we  would  have  listed  ten 
years  ago.  The  review  I  have  made 
so  far  should  make  it  clear  that  edu- 
cational evaluators  can  no  longer 
complain  that  they  do  not  have 
enough  money  or  people,  or  that 
they  are  not  taken  seriously  by  p"1 
icy  makers  and  legislators.  More- 
over, I  do  not  agree  with  those  who 


argue  that  methodological  inade- 
quacies of  one  sort  or  another  pre- 
sent a  major  obstacle  to  the  full 
flowering  of  educational  evaluation 
as  a  policy  instrument.  It  is  not  un- 
common for  social  scientists  to  dis- 
play handw  ringing  despair  over 
their  primitive  methods  and  insen- 
siii\e  measuring  instruments,  and  to 
plead  that  an  Einsteinian  break- 
through in  the  social  sciences  is 
needed  to  put  things  right.  My- own 
view  is  that  we  have  a  long  way  fo 
go  in  making  full  use  of  the  tech- 
niques we  now  have  before  we  are 
in  a  position  to  complain  about  in- 
adequate methods. 

The  newer  problems  education 
evaluators  face  are  of  a  different 
order: 

1.  As  educational  research  and 
e\aluation  have  proliferated,  the 
people  and  institutions  who  are  the 
objects  of  these  studies  have  come 
under  an  increasing  data  collection 
burden  — and  are  increasingly  ex- 
pressing their  resistance  to  it.  It  is 
no  longer  possible  to  assemble  a 
battery  of  interview  schedules  and 
questionnaires  and  invade  the 
schools.  Extensive  prior  clearance 
and  review  are  now  required  almost 
everywhere,  and  outright  refusal  to 
participate  in  studies  is  not  uncom- 
mon The  research  and  evaluation 
community  is  going  to  have  to  work 
out  some  collective  way  of  dealing 
with  this  problem,  for  it  is  a  real 
one.  By  the  time  school  systems 
total  up  all  the  data  collection  re- 
quirements from  Federal.  5tate. 
local,  and  private  sources,  the  bur- 
den often  is  a  crushing  one. 

2.  Evaluation  studies  that  involve 
collecting  data  on  adu::  are  en- 
countering increasing  resistance  at 
the  interviewee  level,  particularly 
among  minorities  and  the  poor 
where  it  is  now  not  uncommon  for 
respondents  to  insist  that  they  be 
paid  for  their  time. 

3.  The  increased  sensitivity  to 
evaluation  studies  — both  what  they 
seek  to  find  out  and  the  amount  of 
data  they  propose  to  collect  — is  re- 
sulting in  a  strangling  growth  of 
revifews,  clearances,  and  advisory 
bodie«.  The  problem  these  mul""1.' 
involvements  and  clearances  pose 
«■-,-  lne  evaluator  are  so  great  il  •  < 
threaten  to  prevent  many  evalua- 
tions from  being  carried  out  at  all. 


4.  As  protests  over  evaluations 
arise,  ostensibly  over  the  type  and 
amount  of  data  to  be  collected,  there 
is  likely  to  be  an  increased  politici- 
zation  of  these  protests  and  their  use 
as  weapons  in  broader  disputes  be- 
tween local  and  Federal  levels  of 
government. 

5.  As  evaluation  activity  and  pol- 
icymakers' interest  in  it  have  grown, 
there  has  also  been  an  increased 
awareness  at  the  program  level  that 
it  is  necessary  to  start  taking  evalu- 
ations seriously.  This  has  had  the 
unfortunate  effect  on  some  program 
officers  and  school  administrators 
of  increasing  their  unwillingness  to 
participate  in  evaluation  studies  for 
fear  of  what  will  happen  to  their 
programs  if  the  evaluation  produces 
negative  findings. 

6.  Evaluations  are  increasingly 
encountering  unrealistic  expectat- 
ions on  the  part  of  policymakers 
and  legislators  with  respect  to  both 
the  speed  with  which  evaluations 
should  be  mounted  and  completed, 
and  the  simplicity  of  the  answers 
which  are  desired.  A  demand  for 
evaluation  has  been  created,  and  it 
is  an  increasingly  insistent  one.  Pol- 
icymakers are  beginning  to  display 
an  irritated  impatience  with  the 
elaborate  trappings  of  careful  de- 
-:-;n,  longitudinal  studies,  and  com- 
plex multivariate  findings.  Ju"\ 
want  to  know  whether  or  not  a  pro- 
gram is  any  good  and  they  want  to 
know  it  yesterday.  As  unrealistic  as 
these  expectations  are.  evaluators 
tnemselves  probably  must  bear 
some  of  the  blame  for  them.  In  their 

rly  zeal  to  have  the  virtues  of 
evaluation  recognized  and  used, 
evaluators  were  almost  certainly 
guilty  of  overpromising. 

This  problem  has  already  passed 
the  stage  of  irritated  impatience. 
Last  year,  the  Congress  cut  the  Of- 
fice of  Education's  evaluation  budget 
in  half  and  made  large  reductions  in 
its  statistics  budget  and  in  NIE's 
research  funds. 

7.  We  are  certain  to  see  a  lot 
more  public  debate  of  the  kind  1 
noted  earlier  over  the  validity  of 
evaluation  methods  and  results.  \n 
increasingly  important  and  .ime 
consuming  task  for  evaluators  will 
be  defending  the  evaluu'ions  they 
carry  out  and  their  suitability  as  a 
basis  for  policy  decisions.  An  unfor- 
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tunale  by-product  of  such  debates  is 
the  impression  created  among  both 
policy-makers  and  the  public  that 
the  mere  fact  such  a  debate  is  oc- 
curring means  the  evaluation  must 
ipso  Jaclo  be  faulty  and  therefore 
should  be  put  aside.  It  is  ironic  that 
after  a  large  scale  formal  evaluation 
has  been  put  aside  because  of  tech- 
nical questions  raised  about  its 
methodology,  policymakers  and  pro- 
gram officials  then  return  to  the  old 
and  familiar  methods  of  making  the 
decision  or  formulating  the  policy- 
methods  that  are  highly  partisan 
and  subjective  in  nature. 

The  seriousness  of  these  problems 
should  not  be  underestimated  mere- 
ly because  they  are  technical  and 
procedural  in  character.  Perhaps 
we  can  take  some  solace,  however, 
in  the  realization  that  these  are  the 
problems  of  impact  and  success 
rather  than  the  problems  of  neglect 
and  disregard. 

The  fact  that  such  problems  are 
being  encountered  is  an  indication 
of  how  far  evaluation  has  come  in 
the   last   decade.    Educational  eval- 


uation has  gone  from  not  being 
taken  seriously  to  being  expected  to 
produce.  It  has  gone  from  a  condi- 
tion of  no  funds,  people,  or  influence 
to  one  of  being  held  accountable  for 
producing  valid  and  useful  studies. 
It  has  gone  from  not  having  enough 
money  to  do  evaluations  to  the 
technical  and  political  problems  of 
carrying  them  out.  Some  evaluators 
who  have  struggled  so  hard,  to 
bring  about  these  changes  are  noV 
wistfully  wondering  whether  they 
wouldn't  just  as  soon  have  their  old 
problems  back.  As  Oscar  Wilde  ob- 
served, there  are  two  particularly 
dissatisfy ing  things  in  life: the  first 
is  not  getting  what  you  want;  the 
second  is  getting  it. 

In  sum,  while  I  do  not  agree  with 
the  cynical  view  which  holds  that 
educational  evaluation  is  largely  a 
waste  of  time  because  its  methods 
are  too  weak,  because  it  will  be  for- 
ever unsupported,  or  because  im- 
portant policies  and  decisions  will 
be  made  in  spile  of  evaluation  find- 
ings, and  while  I  believe  that  im- 
portant progress  has  been  made  in 


educational  evaluation  during  the 
past  decade,  I  nevertheless  believe 
that  educational  evaluation  now 
faces  a  new  array  of  problems  that 
are  possibly  more  serious  than  the 
basic  ones  of  getting  the  necessary 
resources  to  do.  evaluations.  These 
new  problems  are  a  strange  mix- 
ture of  logistics  and  politics,  and 
they  are  in  large  part  an  outgorwth 
of  the  incteasing  pluralism  of  Amer- 
ican society.  If  they  are  not  dealt 
•with,  evaluation  will  not  succeed  in 
making  more  than  an  occasional  or 
marginal  impact  on  educational 
policies  and  programs.  If  they  can 
be  solved,  the  general  trend,  which 
has  only  recently  been  established, 
can  be  continued;  and  the  wider  use 
of  evaluation  can  make  a  major  con- 
tribution to  the  setting  of  national 
educational  policies,  to  the  develop- 
ment of  education  programs,  and  to 
the  allocation  of  scarce  educational 
resources. 
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DISCUSS1CH 

Several  methodological  and  organizational  issues  were  raised 
following  Dr.  Evans'  presentation.   Like  other  speakers,  he  said 
that  effective  use  of  evaluation  information  requires  that  the  re- 
search be  sponsored  by  an  office  located  strategically  to  influence 
the  agency's  planning  and  budgeting  processes.   It  is  important  for 
methodological  expertise  to  be  built  into  the  agency.   However  agen- 
cies should  have  staff  capable  of  overseeing  the  methodological  rigor 
of  the  study,  and  not  leave  this  responsibility  to  extramural  re- 
searchers. 

Questions  were  raised  about  the  Office  of  Education's  use  of 
sole  source  contracting  for  evaluation  research.   Dr.  Evans  said 
that  this  practice  has  been  discontinued  and  now  the  agency  uses 
only  competitive  bidding  to  procure  evaluation  research. 

Like  other  speakers,  Dr.  Evans  noted  that  evaluation  is  often 
hampered  by  the  absence  of  precise  legislative  goal  statements.   This 
situation  is  being  improved  since  agency  staff  have  begun  to  estab- 
lish better  working  relationships  with  committee  staff  to  get  a  bet- 
ter idea  of  intended  program  goals . 
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"EVALUATING  EVALUATION:   A  CONGRESSIONAL  PERSPECTIVE,"  ALLEN 
SCHICK,  CONGRESSIONAL  RESEARCH  SERVICE 


As  the  concluding  speaker  at  this  CRS-sponsored  seminar  on  pro- 
gram evaluation  for  Congress,  I  have  the  enviable  task  of  evaluating 
evaluation.   During  the  sessions  of  this  seminar,  we  have  been  told 
why  evaluation  is  a  good  thing  and  what  Congress  is  doing  to  get  more 
of  it.  We  have  been  told  that  the  executive  branch  is  on  its  own 
evaluation  trip  and  that  there  are  new  and  sophisticated  techniques 
for  assessing  the  costs  and  effectiveness  of  Government  programs. 
Everything  seems  to  be  in  GO  condition  for  a  great  leap  forward  for 
the  evaluation  business. 

But  a  bit  of  history  should  caution  us  that  things  might  not  be 
so  easy.   Although  the  Legislative  Reorganization  Act  of  1946  canon- 
ized "continuous  watchfulness"  as  the  duty  of  all  congressional  com- 
mittees, legislative  oversight  still  is  not  practiced  in  a  regular  or 
comprehensive  fashion.   Thirty  years  after  the  1946  act,  the  promise 
of  legislative  oversight  is  far  from  realization  and  one  cannot  be 
sure  that  the  impetus  of  the  1974  Congressional  Budget  Act  and  the 
House  Committee  Amendments  will  suffice  to  overcome  the  lack  of  per- 
sistent, indigeneous  incentives  for  evaluation. 

There  must  be  a  reason  why  evaluation  is  so  wanted  and  so  unat- 
tained.  My  own  hunch  is  that  not  having  had  to  do  with  the  wanting, 
that  program  evaluation  is  coveted  because  there  is  so  little  of  it. 
The  prize  has  become  more  valued  as  the  stakes  have  escalated  and  as 
disappointments  with  program  results  have  increased.  Thus,  within 
the  space  of  30  years,  Federal  expenditures  will  have  climbed  tenfol 
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from  approximately  $35  billion  after  World  War  II  to  the  $350  billion 
plus  projected  for  the  bicentennial  fiscal  year. 

The  Executive  Is  a  Not  So  Evaluating  Branch 

Self -evaluating  organizations  are  administrative  freaks  ar.i  t'r.ey 
rarely  rank  among  the  more  successful  or  durable  executive  agencies. 
From  time  to  time,  organizations  invest  in  stocktaking  and  retrospec- 
tion, but  these  tend  to  be  mere  pauses  in  the  striving  for  security 
and  growth.   The  exceptions  shed  some  light  on  the  normal  impulse  of 
self-promotion.   During  the  sad  decade  of  its  short  life,  the  Office 
of  Economic  Opportunity  (OEO)  excelled  in  its  sponsorship  and  utiliza- 
tion of  program  evaluations,  even  to  the  extent  of  terminating  some  of 
its  most  promising  initiatives  when  they  proved  to  be  ineffective.  In 
OEO,  evaluation  came  to  be  a  substitute  for  action,  a  manner  of  exis- 
tence appropriate  for  a  mendicant  rather  than  an  affluent  organization. 
Lacking  sufficient  resources  to  underwrite  regular  operating  programs, 
OEO  applied  its  meager  resources  to  small-scale  experiments  and  pilot 
tests  in  a  futile  effort  to  stimulate  social  innovation.   However, 
well-funded  agencies  tend  to  move  into  action  before  they  have  evalu- 
ative confirmation  of  the  worthwhileness  of  their  program  innovations, 
and  they  continue  what  they  are  doing  even  if  the  results  are  unfav- 
orable. 

There  are  a  number  of  reasons  why  agencies  do  not  regularly  eval- 
uate their  own  performance. 

(1)  The  Budget  Problem.   The  annual  budget  process  would  seem  to 
offer  a  hospitable  opportunity  for  assessing  results  and  applying  them 
to  next  year's  money  decisions.   On  paper  at  least,  each  year  an  agency 
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has  to  be  born  anew  and  its  course  can  be  redirected  by  budgetary 
choices.   Yet  the  straight  fact  is  that  budgeting  is  ill-suited  for 
an  evaluation  role  and  that  in  many  environments  evaluation  might 
have  a  better  chance  if  it  is  separated  from  the  routines  of  bud- 
geting. 

Agencies  are  not  neutrals  in  the  recurring  budgetary  wars;  they 
are  budget  maximizers.   The  language  of  budgeting  is  geared  to  ad- 
vocacy and  justification,  not  to  the  objective  search  for  truth.  Rath- 
er than  face  a  life  or  death  situation  every  year,  agencies  confront 
the  budget  with  expectations  of  incremental — or  better — growth  and 
with  confidence  that  their  established  level  of  activities  and  fund- 
ing is  secure.   Because  they  almost  always  receive  at  least  as  much 
money  as  they  were  given  for  the  previous  year,  agencies  have  a  weak 
incentive  to  review  what  they  have  been  and  to  seek  more  useful  ac- 
tivities.  In  the  context  of  the  annual  budget  struggle,  a  truly 
objective  evaluation  can  injure  an  agency's  budget  position  by  in- 
viting outsiders  (such  as  Congress)  to  reduce  appropriations  below 
previously  fixed  levels.   If  an  agency  deploys  its  evaluations  to 
substitute  new  for  existing  programs,  it  runs  the  risk  of  losing  its 
existing  programs  without  getting  the  new  ones.   A  case  in  point  is 
the  current  effort  by  the  Secretary  of  Defense  to  utilize  savings 
from  reductions  in  headquarters  overhead  for  additional  Army  divi- 
sions.  If  Congress  decides  to  convert  the  savings  into  fewer  mili- 
tary dollars  rather  than  more  firepower,  it  will  be  difficult  for 
the  Defense  Department  to  sustain  this  evaluation  effort. 
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The  pace  of  budgeting  discourages  the  application  of  evaluative 
findings.   Budgeting  is  among  the  most  routine  of  administrative  pro- 
cesses, responding  with  cyclical  regularity  to  timetables  and  dead- 
lines.  It  is  not  easy  to  cram  evaluation  into  this  fixed  schedule 
of  events. 

(2)  Evaluation  is  an  Unpleasant  Task.   Evaluation  is  not  the  most 
effective  way  to  make  friends  and  influence  people  within  your  own  or- 
ganization.  An  evaluator  quickly  becomes  an  outsider  within  his  own 
organization,  one  whose  views  and  loyalty  are  suspect.   There  is  good 
reason  why  evaluators  are  guaranteed  in  "inspector  general"  units  and 
why  the  chief  auditor  of  the  United  States  is  graced  with  a  15-year 
term.   With  normal  participation  in  organization  activities,  the  pros- 
pect is  that  an  evaluator  will  be  quickly  seduced  by  the  ethic  and 
striving  of  the  organization.   It  is  not  so  much  a  matter  of  a  person 
tolerating  inefficient  or  unproductive  work;  rather,  most  people  tend 
to  embrace  the  identities  and  objectives  set  by  their  organizations. 

(3)  An  Organization  is  a  Captive  of  its  Clients.   Once  a  program 
is  established  and  benefiting  its  clients,  an  agency  cannot  easily  re- 
direct its  efforts.   With  their  particular  view  of  a  program's  bene- 
fits, the  clients  may  have  a  different  assessment  than  one  emerging 
from  more  objective  criteria  of  costs  and  benefits.   There  is  an  un- 
derstandable tendency  for  beneficiaries  to  believe  that  what's  good 
for  them  is  good  for  the  U.S.A.   Even  comparatively  new  programs 
quickly  fall  prey  to  capture  by  their  clients,  for  once  a  program 

is  underway,  the  benefits  are  "vested"  in  their  recipients  and  hard 
to  terminate.   This  is  one  of  the  reasons  why  persons  interested  in 
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program  evaluation  tend  to  concentrate  on  pilot  tests  and  experiments 
prior  to  full  operationalization  of  the  program. 

(4)  Evaluation  is  a  Difficult  Task.   Although  there  have  been  sig- 
nificant recent  advances  in  evaluation  methodology — for  example,  in  the 
construction  of  samples  and  control  groups — the  routinization  of  eval- 
uation remains  a  distant  possibility.   Part  of  the  problem  is  that  eval- 
uative data  are  difficult  and  costly  to  secure;  they  are  not  ordinarily 
generated  as  a  matter  of  organizational  routine.   Unlike  conventional 
management  information  which  relates  to  the  internal  operations  of  the 
agency,  evaluative  data  frequently  deal  with  what  is  happening  outside 
the  organization,  and,  hence,  it  takes  a  special  effort — such  as  a  sur- 
vey or  experimental  activities — to  produce  the  wanted  data.   For  example, 
a  well-run  health  organization  is  likely  to  possess  a  great  deal  of  work- 
load data  such  as  patient-staff  ratios,  per  patient  costs,  and  number  of 
meals  served,  but  it  is  not  as  likely  to  be  informed  about  the  morbidity 
and  mortality  rates  of  the  population  it  serves. 

Even  when  data  have  been  secured,  their  interpretation  poses  nu- 
merous difficulties.   Rarely  are  the  findings  so  unambiguous  as  to  per- 
mit only  one  interpretation.   More  commonly  the  data  are  open  to  various 
interpretations,  depending  on  the  assumptions  that  are  used  and  the  ob- 
jectives against  which  the  results  are  measured.   As  the  famous  Head 
Start  evaluations  demonstrated,  even  a  costly  evaluation  is  vulnerable 
to  methodological  attack  by  persons  who  refuse  to  accept  the  official 
interpretations.   A  decade  later,  scholars  and  practitioners  alike  still 
are  warring  over  the  soundness  and  policy  implications  of  the  Coleman 
report. 
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(5)  Fear  of  Not  Flying.   I  have  a  feeling  that  organizations  would 
be  more  receptive  to  evaluations  if  they  proved  the  worthwhileness  of 
the  established  programs.   Evaluation  in  the  service  of  program  advo- 
cacy would  be  an  effective  tool  for  agencies  seeking  higher  budgets 
and  support  for  their  preferred  programs.   Yet  many  of  the  evaluations 
undertaken  during  the  past  decade  have  cast  serious  doubts  on  the  cost- 
effectiveness  of  major  domestic  programs.   For  every  evaluation  that 
highlights  program  success,  there  are  handfuls  which  pronounce  failure 
or  uncertain  results.   This  has  been  the  sad  fate  of  many  of  the  housing 
and  education  innovations  launched  so  enthusiastically  in  the  1960's. 
The  market  for  evaluation  dries  up  when  evaluation  becomes  an  anti-or- 
ganization weapon. 

Worse  yet,  evaluation  has  been  concentrated  on  program  innovations 
rather  than  on  long  established  activities.   Thus,  Title  I  ESEA  has  been 
evaluated  in  dozens  of  communities  from  dozens  of  perspectives  while  the 
basic  education  programs  which  claim  tens  of  billions  of  dollars  have  es- 
caped serious  scrutiny.   This  means  that  the  most  venturesome,  riskiest, 
activities  are  evaluated  while,  safe  and  perhaps  unproductive  programs 
are  not  subjected  to  the  same  tests.   Strangulation  by  evaluation  has 
been  the  dismal  fate  of  many  of  the  exciting  education  innovations  of 
recent  years  such  as  performance* contracting  and  school  vouchers.   If 
evaluation  is  to  succeed,  it  must  be  applied  to  old  and  new  programs  and 
must  not  be  deployed  as  a  weapon  against  program  change.  An  anti-in- 
novation bias  was  inherent  in  the  congressional  budget  reform  bill  for- 
mulated in  1973  by  the  Senate  Government  Operations  Committee.   As  re- 
ported, new  programs  could  not  be  implemented  until  they  had  undergone 
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mandatory  pilot  testing  and  evaluation.   This  bias  was  removed  in 
later  versions  of  the  legislation. 

When  an  agency  wants  a  new  program,  it  might  try  to  disregard 
or  discredit  an  adverse  evaluation.   Such  was  the  case  in  the  early 
1970' s  when  the  Defense  Department  promulgated  a  "fly  before  you  buy1 
policy  for  major  weapons  procurement.   But  shortly  after  a  prototype 
F-14  crashed,  the  Navy  rushed  to  order  a  substantial  number  of  the 
new  planes. 

(6)  Inadequate  Congressional  Demand  for  Evaluation.  In  view  of 
the  powerful  disincentives  within  executive  agencies,  the  pressure 
for  evaluation  must  come — if  at  all — from  outside  sources.   Yet  the 
weak  agency  commitment  to  evaluation  is  reinforced  by  lack  of  con- 
gressional interest.   Congress  rarely  insists  that  agencies  system- 
atically monitor  their  own  performance  and  report  the  findings  at 
hearings  or  other  legislative  forums.   At  least  two  explanations  may 
be  offered  for  congressional  disinterest:   first,  committees  may 
function  as  program  advocates  as  much  as  program  overseers;  second 
Congress  tends  to  be  more  oriented  to  prospective  legislation  than 
to  past  enactments. 

Authorizing  committees  may  become  advocates  for  the  agencies 
and  programs  they  are  charged  to  oversee.   Sometimes  this  is  due 
to  the  "capture"  of  the  committee  by  the  agency,  a  process  which  is 
fomented  by  close  and  continuing  contact  between  agency  and  commit- 
tee staffs  and  by  the  understandable  inclination  of  congressmen  to 
identify  with  the  missions  and  activities  of  the  agencies  under 
their  purview.   Still  another  factor  is  that  the  authorizing  role 
often  casts  the  committee  in  an  adversary  relationship,  vis  a  vis 
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the  appropriations  committees  with  the  result  that  the  authorizers 
feel  impelled  to  support  the  agencies  under  their  jurisdiction. 

As  a  legislating  institution,  Congress  tends  to  rate  its  success 
and  effectiveness  in  terms  of  the  bills  it  passes  rather  than  in  terms 
of  agency  fidelity  to  earlier  legislation.   Congress  is  oriented  to 
the  future,  not  to  the  past,  so  that  there  is  a  chronic  neglect  of 
its  oversight  role. 

The  Role  of  Congress 

Yet  oversight  is  not  a  new  role  for  Congress  but  one  fully  in 
accord  with  its  generic  purpose  as  a  check  on  executive  authority. 
With  the  growth  of  big  Government,  and  in  particular  big  bureaucracy, 
the  importance  of  this  role  is  greater  than  ever  before.   Inadequate 
attention  to  oversight  cannot  be  blamed  on  inadequate  staffing,  though 
there  surely  remain  critical  staff  shortages  in  certain  areas.   The 
evidence,  however,  is  that  the  enlargement  of  staff  and  other  re- 
sources during  the  past  decade  has  not  yet  produced  a  breakthrough 
with  regard  to  oversight  by  Congress.   In  comparing  congressional 
capability  versus  practice,  one  is  reminded  of  the  epitaph  on  the 
tombstone  of  an  athiest,  "All  dressed  up  and  nowhere  to  go."  If  Con- 
gress is  on  the  brink  of  a  revised  interest  in  legislative  oversight, 
part  of  the  reason  is  that  for  the  first  time  it  is  tooled  up  to  per- 
form this  role  in  an  effective  manner. 

Some  of  the  recent  steps  to  bolster  the  oversight  function  have 
been  mentioned  by  earlier  speakers  and  these  include  the  designation 
of  oversight  subcommittees  in  the  House,  expansion  of  GAO's  role  in 
program  evaluation,  and  the  new  congressional  budget  process.   None 
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of  these  laudable  moves  ensures  the  success  of  legislative  oversight, 
but  they  at  least  remove  some  of  the  impediments  to  this  role. 

It  should  be  noted  that  I  have  been  using  the  term  "program  over- 
sight" in  referring  to  executive  branch  activities  and  "legislative 
oversight"  when  referring  to  Congress.   This  distinction  is  predicated 
on  a  division  of  labor  between  the  two  branches,  with  executive  agen- 
cies carrying  the  brunt  of  evaluation  efforts  and  Congress  overseeing 
agency  performance.   There  is  not  much  sense  in  trying  to  make  Con- 
gress into  an  evaluating  institution,  though  on  occasion  Congress  may 
perform  this  function.   Evaluation  should  not  be  a  sometime  activity, 
conducted  according  to  the  inevitable  spasms  of  legislative  interest, 
but  a  regular  part  of  administrative  implementation  of  congress ionally- 
authorized  programs.   The  appropriate  role  of  Congress  is  to  monitor 
performance  and  to  prod  agencies  to  evaluate  their  own  activities.   To- 
day a  1946  "continuous  watchfulness"  remains  the  essence  of  legisla- 
tive oversight.   Perhaps  the  biggest  payoff  from  a  more  independent 
and  vigorous  oversight  role  would  be  to  stimulate  more  and  better 
evaluations  by  executive  agencies.   Consequently,  in  the  concluding 
portion  of  my  talk,  I  will  identify  some  ways  in  which  Congress  can 
goad  agencies  to  perform  useful  evaluations. 

(1)  Identification  of  Objectives.   Despite  all  the  emphasis  on 
early  legislative  specification  of  objectives  and  intent,  there  is  a 
limit  as  to  how  much  precision  and  clarity  can  be  achieved  in  com- 
plicated and  controversial  legislation.   Legislators  often  are  pro- 
voked by  a  sense  of  a  problem,  not  by  some  firm  objective.   They 
perceive  something  awry,  are  moved  to  legislative's  remedy,  but  do 
not  have  a  precise  notion  of  what  the  future  should  be,  except  that 
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they  want  the  problem  eliminated.   When  legislation  is  controversial 
and  objectives  clash,  there  is  a  tendency  to  paper  over  differences 
by  being  vague  about  purposes  and  intent.   Thus,  parties  with  differing 
objectives  are  able  to  subscribe  to  the  same  legislative  package. 

When  Congress  muddles  through  without  a  clear  specification  of 
purpose,  the  process  of  evaluation  is  not  aborted.   Rather,  it  must 
begin  with  executive  implementation  rather  than  with  textbook  clarity 
about  objectives.   Congress,  in  fact,  has  two  legitimate  tasks  to  get 
the  Executive  to  produce  relevant  evaluations.   First,  Congress  can 
write  into  law  a  mandate  for  the  agency  to  evaluate  its  program;  sec- 
ond, Congress  can  demand  that  the  evaluative  measures  used  by  the  agen- 
cy reflect  congressional  interest  and  perspectives,  not  merely  the 
orientation  of  the  implementing  agency.   Statutory  set  asides  of  funds 
for  evaluation  do  not  suffice  to  establish  an  adequate  evaluation  pro- 
cedure.  What  is  required  is  the  prescription  in  law  of  a  process  that 
agencies  must  adhere  to  in  evaluating  their  activities,  including 
milestones  for  the  crucial  events  in  the  process,  and  a  reporting 
schedule. 

(2)  Keep  Evaluations  Honest.   If  self-evaluation  is  difficult,  one 
should  not  be  surprised  by  the  encroachment  of  organizational  bias  into 
even  sophisticated  evaluations.   One  form  of  bias  arises  out  of  the  self- 
protection  practiced  by  agencies  to  avert  unfavorable  findings.   The 
best  antidote  for  this  kind  of  mischief  is  to  "let  it  all  hang  out," 
assumptions  and  all,  so  that  the  judicious  reader  can  determine  the 
validity  of  the  results.   A  more  formidable  kind  of  bias  arises  out 
of  differing  executive  and  legislative  perspectives,  in  particular  the 
holistic  viewpoint  of  administrators  versus  the  specific  interests  of 
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congressmen.   Congress  might  be  more  concerned  about  the  distributional 
effects  of  a  program — who  gets  what — than  about  the  impact  on  net  na- 
tional welfare.   However,  congressional  interest  in  program  impact  on 
specific  target  groups  might  not  be  reflected  in  the  design  of  an  eval- 
uation unless  there  is  adequate  legislative  input  at  the  start.   One  of 
the  functions  of  the  new  oversight  subcommittees,  therefore,  might  be 
to  examine  prospective  evaluations  to  gauge  their  sensitivity  to  the 
distribution  of  costs  and  benefits  among  various  groups. 

(3)  Expand  the  Time  Framework.   The  time  perspective  of  most  eval- 
uations is  notoriously  short  and  there  is  real  danger  that  the  pace  of 
legislative  politics  might  induce  agencies  to  work  with  even  narrower 
time  spans.   Typically,  an  education  experiment  is  tested  during  the 
ten  months  of  the  school  year  and  if  results  are  not  forthcoming,  the 
experiment  is  terminated.   The  demand  is  for  quick  results,  to  over- 
come within  a  few  months  the  handicaps  of  years  and  generations.   It 
is  not  surprising  that  so  many  evaluations  show  failure  for  programs 
inaugurated  with  promise  and  fanfare  only  months  earlier. 

Patience  is  very  much  in  short  supply  in  program  evaluation,  not 
only  because  longitudinal  studies  are  costly  and  difficult  to  engineer 
but  also  because  administrators  and  legislators  have  a  "now"  orienta- 
tion and  are  unwilling  to  forego  immediate  gain  in  the  quest  for  fu- 
ture improvement . 

At  the  present  time,  the  technology  of  social  evaluation  is  far 
more  advanced  than  our  understanding  of  basic  social  processes.   We 
are  bewildered  when  some  Title  I  evaluations  show  success  while  others 
point  only  to  failure,  as  if  there  were  random  or  unknown  factors  at 
work.   The  real  problem  is  that  we  have  been  willing  to  invest  in 
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social  experimentation  but  not  in  social  understanding.  At  this 
juncture,  it  might  be  wiser  to  commission  long-term  studies  than 
to  finance  more  quickie  research.   But  unless  Congress  is  willing 
to  tolerate  the  long  view,  agencies  will  continue  to  engage  in  LIFO- 
type  evaluations,  with  the  last  program  started,  the  first  one  ter- 
minated. 

(4)  Make  Them  Readable.   Many  of  the  recent  methodological  gains 
in  the  state  of  the  art  have  contributed  to  an  escalation  in  the  com- 
plexity and  obscurantism  of  evaluations.   Perhaps  the  most  difficult 
task  in  evaluation  is  to  translate  quality  research  into  readable 
policy  analysis.   The  analyst  often  is  insensitive  to  the  importance 
of  this  task  or  incapable  of  escaping  the  constraints  of  his  adopted 
language,  but  Congress  need  not  suffer  incomprehension  gladly.   Con- 
gress should  insist  that  the  fruits  of  evaluation  be  communicated  in 
a  straightforward  and  understandable  way,  that  jargon  and  sophistry 
not  be  made  into  barriers  to  sound  public  policy.   A  side  benefit 
might  be  to  expose  faulty  assumptions,  inadequate  data,  and  question- 
able interpretations.   When  an  evaluation  is  chock  full  of  mumble 
jumble,  the  advantage  is  all  on  the  side  of  the  analyst,  and  Congress 
cannot  exercise  much  independent  judgment.   Readable  evaluations  are 
indispensable  if  evaluation  is  not  to  become  an  executive  violation 
of  legislative  power  and  independence. 

(5)  Input  into  the  Legislative  Process.   There  is  no  special 
value  in  evaluation;  the  payoff  comes  in  using  evaluative  findings 
for  program  design  and  implementation.   Yet,  there  often  is  little 
follow-up  to  an  evaluation;  once  done,  the  case  is  closed  and  the 
evaluators  move  on  to  other  matters.   Each  evaluation  is  regarded 


353 


as  a  discrete,  special  event  rather  than  as  part  of  an  overall 
policy  process. 

Congress  has  a  vital  role  in  encouraging  the  application  of 
analysis  to  public  policy.   Its  two  key  points  of  input  are  the 
authorization  and  appropriations  processes,  and  ray  expectation  is 
that  neither  presents  a  perfectly  hospitable  environment  at  the 
present  time.   As  discussed  earlier,  the  impulse  for  authorizers 
to  double  as  program  advocates  impairs  their  ability  to  function 
as  program  evaluators .   With  regard  to  the  appropriations  process, 
it  seems  as  targetted  on  incremental  patterns  as  executive  branch 
budgeting.   Yet  there  is  some  possibility  that  the  new  congressional 
budget  process  might  bolster  the  incentives  for  evaluation  at  these 
pivotal  legislative  stages,  if  only  because  of  the  pressure  to  con- 
sider particular  spending  demands  in  the  light  of  overall  national 
priorities  and  other  claims  on  the  budget.   Moreover,  the  new  con- 
gressional process  might  contribute  to  a  narrowing  of  the  authori- 
zations-appropriations gap,  the  proclivity  to  authorize  one  level 
of  expenditure  but  to  appropriate  at  a  much  lower  level.   More 
realistic  authorizations  might  encourage  committees  to  take  more 
careful  looks  at  the  programs  subject  to  their  oversight. 

As  long  as  each  evaluation  is  an  "event,"  a  special  occasion 
that  commands  special  notice  and  applause,  we  can  be  sure  that  the 
actual  progress  of  evaluation  in  terms  of  its  utilization  for  pub- 
lic choice  is  inadequate.  Only  when  evaluation  becomes  a  regular 
part  of  program  implementation  will  it  recede  as  something  special 
and  become  only  something  useful  for  Government.  By  this  test,  we 
all  have  a  long  way  to  go. 
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LETTER  OF  TRANSMITTAL 


March  14,  1975. 
Hon.  Carl  Albert, 

Speaker  of  the  House  of  Representatives, 
Washington  D.C. 

Dear  Mr.  Speaker:  Pursuant  to  rule  18(b)  of  the  Committee  on 
Government  Operations,  I  submit  herewith  the  committee's  report  to 
the  94th  Congress,  as  required  by  House  rule  X,  2(c)  relating  to  the 
oversight  plans  of  the  standing  committees  of  the  House  of 
Representatives. 

Jack  Brooks,  Chairman. 
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OVERSIGHT  PLANS  OF  THE  COMMITTEES  OF  THE 
U.S.  HOUSE  OF  REPRESENTATIVES 


Introduction 


In  adopting  the  Committee  Reform  Amendments  of  1974,  the  93d 
Congress  recognized  there  were  serious  shortcomings  in  what  is  one 
of  the  most  important  congressional  functions — oversight  of  Federal 
programs  and  agencies. 

The  requirement  first  set  forth  in  the  Legislative  Reorganization 
Act  of  1946  that  each  standing  committee  exercise  a  continuing  study 
and  review  of  the  statutes  under  its  jurisdiction  and  the  executive 
agencies  and  departments  responsible  for  administering  them,  was 
made  far  more  explicit  by  the  93d  Congress. 

In  a  further  effort  to  strengthen  the  overall  oversight  function  of 
the  House,  it  included  in  the  reform  amendments  a  provision  now  in- 
corporated in  clause  2(c)  of  rule  X,  which  reads  as  follows: 

At  the  beginning  of  each  Congress  an  appropriate  repre- 
sentative of  the  Committee  on  Government  Operations  shall 
meet  with  appropriate  representatives  of  each  of  the  other 
committees  of  the  House  to  discuss  the  oversight  plans  of  such 
committees  and  to  assist  in  coordinating  all  of  the  oversight 
activities  of  the  House  during  such  Congress.  Within  60  days 
after  the  Congress  convenes,  the  Committee  on  Government 
Operations  shall  report  to  the  House  the  results  of  such  meet- 
ings and  discussions  and  any  recommendations  which  it  may 
have  to  assure  the  most  effective  coordination  of  such  activi- 
ties and  otherwise  achieve  the  objectives  of  this  clause. 
(3) 
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Background 

As  one  of  the  weights  Congress  can  drop  on  the  scale  in  an  effort  to 
achieve  a  balance  between  the  legislative  and  executive  branches,  over- 
sight has  assumed  an  increasingly  important  role  in  recent  years.  This 
trend  reflects  a  widespread  concern  in  Congress  that  the  growth  of  the 
Federal  Establishment  has  outstripped  congressional  efforts  to  super- 
vise it.  The  chilling  example  of  unchecked  Federal  power  afforded  by 
"Watergate-related  issues  should,  alone,  provide  a  sufficient  spur  for 
action. 

But,  oversight  should  not  be  seen  only  as  a  club  held^over  the  execu- 
tive branch.  Ideally,  it  should  complement  executive  management,  pro- 
viding clear  policy  limits  within  which  an  administrator  is  free  to  act. 
The  certain  knowledge  that  failures  of  administration  will  be  taken 
into  account  by  an  alert  oversight  committee  should  produce  the  effi- 
ciency of  operation  that  both  Congress  and  the  executive  branch  desire. 

Oversight,  especially  in  recent  years,  has  come  to  be  viewed  primar- 
ily as  an  investigative  activity,  and  it  is  true  investigations  have 
played  a  significant  role  in  helping  Congress  keep  track  of  executive 
branch  operations.  But  there  are  other  ways  of  accomplishing  the 
same  end.  some  more  suitable  to  the  continuing  supervision  that  is 
contemplated  by  the  new  oversight  rules.  Where  the  goal  is  to  improve 
efficiency,  reinforce  policy  guidelines  or  reduce  costs,  rather  than  to 
expose  some  flagrant  misdeed  in  the  executive  branch,  a  conference 
between  committee  and  agency  representatives  can  frequently  achieve 
the  desired  result.  The  authorization  and  appropriation  process  offers 
committees  a  periodic  opportunity  to  review  the  administration  of 
many  departments  and  take  whatever  corrective  action  is  needed. 

Oversight,  in  fact,  is  not  a  structured  procedure  in  the  manner  of 
Congress  legislative  function.  A  continuously  watchful  eye  is  the  main 
requirement  for  determining  whether  an  executive  agency  is  perform- 
ing a?  it  should,  backed  by  a  willingness  to  act  when  it  is  not. 

Although  practiced  in  a  variety  of  ways  throughout  the  history  of 
the  nation,  oversight  was  not  formally  authorized  as  a  function  of  each 
standing  committee  until  passage  of  the  Legislative  Reorganization 
Act  of  1046.  That  same  act  revived  the  long  defunct  Committee  on 
Expenditures  in  the  Executive  Department  and  gave  it  broad  powers 
to  investigate  all  Federal  agencies  and  departments. 

In  1052.  that  committee  was  renamed  the  Committee  on  Government 
Operations  and  now  it  has  been  given  an  even  broader  mandate  and 
asked  to  help  fashion  a  system  for  a  coordinated  review  of  the  Federal 
Government  by  all  committees. 

The  all-embracing  nature  of  the  Committee  on  Government  Opera- 
tions' oversight  jurisdiction,  which  empowers  it  to  examine  "the  over- 
all economy  and  efficiency  of  government  operations  and  activities.*' 
is  underscored  by  the  special  oversight  provisions  of  the  Committee 
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Heform  Amendments  of  1974.  Clause  4(c)  (2)  of  rule  X  states  that  the 
committee  "may  at  any  time  conduct  investigations  of  any  matter 
without  regard  to  *  *  *  provisions  *  *  *  conferring  jurisdiction 
over  such  matter  upon  another  standing  committee.'' 

The  House  rules  also  instruct  the  other  standing  committees  to  in- 
clude in  their  legislative  reports  a  summary  of  any  oversight  findings 
and  recommendations  the  Committee  on  Government  Operations  may 
have  made  under  its  broad  authority  to  investigate  all  levels  of  the 
Government. 

By  including  these  provisions  in  the  rules,  the  House  has  again 
emphasized  its  acceptance  of  the  principle  of  maintaining  an  oversight 
committee  separate  and  apart  from  the  committees  that  have  responsi- 
bility for  initiating  and  continuing  legislative  programs.  While  tiiese 
legislative  committees  carry  out  their  oversight  responsibilities  in  their 
assigned  areas,  only  the  Government  Operations  Committee  has  the 
total  view  of  the  overall  governmental  process  and  avoids  any  ques- 
tion of  lack  of  objectivity  in  connection  with  its  investigations. 

The  Committee  on  Government  Operations  intends  to  use  its  broad 
grant  of  investigative  power  carefully,  but  believes  that  it  is  a  valuable 
and  essential  tool  in  carrying  out  the  oversight  activities  of  the  House. 
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Oversight  Coordination 

The  Committee  on  Government  Operations  recognizes  the  need  for 
the  various  committees  to  coordinate  their  oversight  activities  and 
accepts  its  responsibilities  under  rule  X,  clause  2(c)  to  assist  them  in 
doing  so.  At  the  same  time,  the  committee  feels  constrained  to  point 
out  that  it  has  neither  the  authority  nor  the  desire  to  police  the  other 
committees  of  the  House  in  the  performance  of  their  legitimate  over- 
sight duties. 

Each  standing  committee  has  a  clearly  assigned  jurisdiction  over 
which  it  is  authorized  by  clause  2(b)  (1)  of  rule  X  to  exercise  general 
oversight.  Some  overlapping  of  jurisdictions  is  unavoidable,  and  the 
Committee  on  Government  Operations  does  not  read  into  clause  2(c) 
any  authority  for  it  to  arbitrate  jurisdictional  disputes  that  may  arise. 
Nor  does  it  interpret  the  clause  to  give  it  any  power  to  supervise  the 
other  committees  to  see  if  they  are  meeting  their  oversight  responsi- 
bilities as  outlined  in  accordance  with  the  provisions  of  clause  2(c). 

The  committee  believes  that  the  objectives  of  the  provision  concern- 
ing oversight  coordination  can  be  met  by — 

(1)  Getting  each  of  the  committees  to  focus  on  its  oversight 
responsibilities  and  develop  some  plana  for  carrying  them  out 
early  in  each  Congress; 

(2)  Indicating  where  areas  of  mutual  interest  or  duplication 
of  effort  between  committees  may  occur ,  and 

(3)  Spotting  areas  in  which  no  committee  has  indicated  plans 
for  conducting  oversight  activities. 

The  committee  recognizes  the  difficulties  and  uncertainties  involved 
in  any  effort  to  set  down  an  oversight  agenda  covering  the  next  2  years. 
It  is  emphasized  that  the  oversight  plans  included  in  this  report  are 
only  the  best  estimate  at  this  time  of  what  the  committees  of  the  House 
will  try  to  accomplish  in  the  94th  Congress. 

The  committee  is  particularly  aware  of  the  difficulties  at  the  begin- 
ning of  the  94th  Congress  when  each  of  the  committees  of  the  House 
has  a  substantial  number  of  new  Members  and  where  the  committees 
are  operating  under  the  Committee  Reform  Amendments  of  1974 
for  the  first  time.  At  the  time  this  report  was  prepared,  a  number  of 
the  committees  had  not  yet  completed  all  of  their  organizational  proc- 
esses, and  this  presented  them  with  particular  problems  in  attempting 
to  predict  their  program  for  the  entire  Congress.  The  Government 
Operations  Committee  appreciates  the  complete  cooperation  and  assist- 
ance given  it  by  each  of  the  committees  in  the  preparation  of  this 
report. 

Under  the  procedure  the  Committee  on  Government  Operations 
elected  to  follow,  its  representatives  met  separately  with  those  of  each 
of  the  other  21  standing  committees,  discussed  the  requirements  :>nd 
purposes  of  clause  2(c),  and  asked  them  to  submit  a  statement  setting 
forth  both  their  general  oversight  jurisdiction  and  their  specific  plans 
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for  this  Congress.  It  is  the  belief  of  the  Committee  on  Government 
Operations  that  the  presentation  of  each  committee's  plan  in  its  own 
words  gives  the  House  the  clearest  picture  of  what  the  committees 
see  as  their  major  oversight  activities,  free  from  any  distortions  or 
inadvertent  misrepresentation  that  might  result  from  efforts  to  inter- 
pret or  evaluate  the  plans. 

As  will  be  noted  from  reading  the  committee  responses,  several  com- 
mittees claim  oversight  responsibility  for  the  same  general  subject 
area.  This  is  entirely  consistent  with  the  legislative  responsibilities  of 
these  committees,  but  it  is  the  hope  of  the  Committee  on  Government 
Operations  that  all  committees,  through  an  examination  of  this  report, 
will  acquaint  themselves  with  the  oversight  plans  of  the  other  com- 
mittees, identify  areas  of  potential  duplication,  and  take  steps  to  keep 
such  duplication  to  a  minimum. 

One  method  of  accomplishing  this  might  be  for  committees  sharing 
jurisdiction  to  notify  each  other  when  any  oversight  activity  is  to  be 
undertaken  and  to  share  any  information  they  may  have  gathered 
about  the  operations  of  a  program.  Investigations  might  be  conducted 
jointly,  staff  resources  might  be  pooled,  requests  for  information  from 
executive  agencies  coordinated,  and  the  services  of  the  General  Ac- 
counting Office  used  jointly. 

Such  steps  could  be  especially  effective  in  the  area  of  factfinding,  as 
distinct  from  matters  of  law  or  policy.  For  example,  the  subject  of 
sugar  pricing  is  of  legitimate  interest  to  a  number  of  committees  be- 
cause of  its  effect  on  foreign  trade,  agriculture  and  the  consumer.  Each 
committee  with  jurisdiction  in  these  areas  would  have  a  different 
reason  for  examining  the  subject,  but  all  might  be  served  by  a  single 
set  of  facts  dealing  with  supply,  suppliers,  and  prices. 

In  implementing  for  the  first  time  these  new  oversight  rules,  the 
committee  is  relying  heavily  on  the  other  standing  committees  to  rec- 
ognize and  meet  their  responsibilities.  It  may  be  that  in  future  years 
the  House  will  want  to  delineate  more  specifically  the  role  it  wants  the 
Committee  on  Government  Operations  to  play  in  this  regard.  At  this 
point,  however,  it  is  the  committee's  hope  that  by  getting  the  21  stand- 
ing committees  to  focus  on  oversight  early  in  each  Congress  and  put 
down  their  plans  for  reviewing  the  programs  under  their  jurisdiction, 
it  has  helped  set  the  House  on  the  course  charted  by  the  Committee 
Reform  Amendments  cf  1974. 

It  is  clear  that  the  main  purpose  behind  the  adoption  of  clause  2(c) 
of  rule  X  is  to  foster  cooperation  between  the  committees  of  the  House 
in  carrying  out  their  oversight  duties.  With  a  conscientious  effort  by 
all  committees,  that  highly  desirable  goal  can  be  reached. 
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Committee  Responses 

The  oversight  plans  of  the  committees  form  an  impressive  catalogue 
of  programs  due  to  come  under  congressional  scrutiny.  It  must  be  rec- 
ognized, however,  that  plans  are  all  they  are.  Oversight,  by  its  nature, 
is  often  a  response  to  the  unusual  and  the  unforeseen.  The  unsettled 
state  of  the  economy,  the  complex  efforts  to  draft  a  national  energy 
program,  and  the  volatile  world  political-military  situation,  all  could 
lead  to  executive  actions  that  will  require  prompt  exercise  by  Congress 
of  its  oversight  responsibility.  In  its  discussions  with  the  other  com- 
mittees, the  Committee  on  Government  Operations  made  it  clear  it 
does  not  expect  them  to  be  held  to  the  plans  outlined  in  this  report. 
They  must  remain  flexible,  able  to  shift  their  oversight  resources 
wherever  they  may  be  needed. 

As  has  been  mentioned  previously,  each  committee  was  requested  to 
provide  for  the  Government  Operations  Committee  a  statement  of  its 
plans  in  writing,  over  the  signature  of  its  chairman.  These  responses 
will  inevitably  vary  in  style  and  detail  as  each  committee  was  urged 
to  express  in  its  own  way  its  views  of  its  responsibilities  and  its  plans 
for  carrying  them  out.  In  each  case,  however,  the  committee  repre- 
sentatives were  encouraged  to  state:  (1)  The  structure  of  the  com- 
mittee for  performing  its  oversight  functions;  (2)  a  description  of 
its  continuing  general  oversight  activities;  and  (3)  and  specific  plans 
at  this  time  for  concentrating  on  particular  programs  under  its 
jurisdiction. 

The  committees  were  asked  to  make  their  responses  brief  in  hopes  of 
obtaining  more  widespread  readership  of  the  final  document.  The  com- 
mittee recognizes  that  if  the  purpose  of  this  rule  is  to  be  accomplished, 
the  report  of  the  committee  must  have  widespread  appeal  and  be  of 
such  form  it  can  be  used  as  a  guide  by  large  numbers  of  persons  with 
varied  interests  including  Members  of  Congress,  congressional  staffs, 
the  media,  and  the  public. 

The  responses  of  the  committees  follow : 
(9) 
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RESPONSES  OF  THE  COMMITTEES 


367 
COMMITTEE  ON  AGRICULTURE 


UJS.  HorsE  of  Representatives, 

Committee  on*  Acricultukfv 
Washington,  D.C.,  February  25, 1975.- 
Hon.  Jack  Brooks, 

Chairman,  Government  Operations  Committee,  House  of  Representa- 
tives, Washington,  D.C. 

My  Dear  Mr.  Chairman  :  This  is  in  reference  to  the  oversight  activi- 
ties contemplated  by  the  Committee  on  Agriculture  during  the  94th 
Congress. 

Pursuant  to  House  Rule  X,  clause  2(c)  and  your  letter  to  me,  mem- 
bers of  our  respective  Committee  staffs  met  last  week  to  survey  the 
scope  of  this  Committee's  activities  during  the  current  Congress. 

This  Committee  intends  to  conduct  oversight  and  discharge  its  in- 
vestigative responsibilities  in  several  ways. 

First,  we  plan  to  rely  on  the  membership  of  the  Subcommittee  on 
Department  Operations,  Investigation  and  Oversight  for  the  bulk  of 
this  work.  The  Committee  has  in  fact  provided  in  its  rules  that  none 
of  our  other  nine  Subcommittees  shall  undertake  a  major  investigation, 
without  the  consent  of  the  Chairman  of  the  full  Committee. 

Second,  all  of  our  Subcommittees  will  conduct  oversight  and  review- 
in  the  consideration  of  legislative  proposals  under  consideration. 

And  finally,  the  full  Committee  will  continue  to  exercise  oversight 
and  review  over  those  agencies  which  are  affected  by  measures  conr- 
sidered  in  the  full  Committee. 

The  areas  we  know  will  be  examined  in  the  94th  Congress  include' 
the  Food  Stamp  Program,  for  which  we  are  now  planning  a  thorough 
review,  sugar  pricing  systems,  crop  disaster  and  the  administration 
and  operation  of  the  Federal  Fungicide,  Rodenticide  and  Insecticide 
Act,  and  the  target  price  and  loan  system  on  major  agricultural  crops. 

Naturally  we  cannot  anticipate  all  the  areas  of  upcoming  oversight 
as  events  yet  to  unfold  will  dictate  that  course  of  action  for  us.  Thus 
we  feel  that  we  are  neither  restricted  nor  committed  to  any  specific 
oversight  activities  at  this  particular  time. 

In  order  to  give  your  Committee  and  the  House  a  better  picture  of 
our  oversight  activities,  I  have  attached  a  copy  of  our  Report  to  the 
House  for  the  93rd  Congress  on  the  subject  (House  Report  93-1654).* 
I  would  draw  special  attention  to  the  jurisdictional  range  and  the  44 
specific  oversight  activities  we  carried  out  in  1973-74  as  set  forth  on 
pages  4,  5,  6, 13  and  14. 

I  hope  this  will  be  helpful  to  you  in  carrying  out  your  duties  under 
House  Rule  X. 
Sincerely, 

Thomas  S.  Foley,  Chairman. 

•Document  available  In  flies  of  the  Committee  on  Government  Operations. 
(13) 
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House  of  Representatives, 
Committee  on  Appropriations, 
Washington,  D.C.,  February  27, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  House  of  Repre- 
sentatices,  Washington,  D.C. 

Dear  Mr.  Chairman:  I  am  writing  in  response  to  your  letter  of 
"February  13  and  your  request  for  information  concerning  the  over- 
sight activities  of  the  Appropriations  Committee  in  the  94th  Congress 
as  required  by  the  new  provisions  of  Clause  2(c)  of  Rule  X  of  the 
House. 

As  you  know,  the  Committee  on  Appropriations  is  specifically  ex- 
empted from  general  legislative  oversight  responsibilities  under 
Clause  2(b)(1)  of  Rule  X.  The  Committee  does,  however,  retain 
the  fiscal  oversight' function  under  Clause  2(b)  {?>)  of  Rule  X  which 
was  formerly  contained  in  Clause  2(b)  of  Rule  XI.  Under  this  provi- 
sion the  Committee  is  authorized  to  conduct  studies  and  examinations 
of  the  executive  departments  and  agencies  as  it  deems  necessary  to 
assist  it  in  deliberations  on  matters  within  its  jurisdiction. 

As  I  understand  the  intent  of  the  new  rule,  it  is  the  responsibility  of 
the  Committee  on  Government  Operations  to  perform  a  coordinating 
function  with  respect  to  the  oversight  activities  of  the  various  standi 
ing  committees  of  the  House  and  make  such  recommendations  as  your 
Committee  may  deem  appropriate  to  promote  the  overall  effectiveness 
of  reviewing  the  ongoing  activities  of  the  federal  establishment.  It  is 
understandable  that  the  rule  applies  differently  to  the  Appropriations 
Committee  than  to  the  legislative  committees. 

The  principal  function  of  the  Appropriations  Committee  under 
Rule  X  of  the  House  is  the  "Appropriations  of  the  revenue  for  the  sup- 
port of  the  Government".  In  executing  its  responsibilities,  the  Com- 
mittee through  its  thirteen  subcommittees,  annually  examines  the 
budget  estimates  of  federal  organizations  contained  in  the  Budget 
of  the  United  States.  The  vast  majority  of  this  activity  is  directed 
at  determining  what  amounts  of  money  will  be  required  to  execute  the. 
various  federal  programs  during  the  forthcoming  year  imder  existing 
legislation.  The  Rules  of  the  House  prohibit  the  appropriation  of 
monies  for  purposes  not  previously  authorized.  Therefore,  the  Com- 
mittee examination  of  budget  requests  typically  incorporates  the  pro- 
cedure of  reviewing  the  execution  of  the  ongoing  programs  as  an 
important  yardstick  in  measuring  appropriation  requirements  for  the 
subsequent  fiscal  year.  This  process  has  substantial  "oversight"  charac- 

(14) 
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t  eristics.  It  would  be  difficult,  if  not  impossible,  to  identify  what  por- 
tion  of  the  Committee's  annual  budget  examination  might  be  classified 
as  "oversight"  in  nature,  but  it  is  considerable. 

During  the  last  session  of  Congress,  the  Committee  conducted  over 
400  days  of  hearings,  generating  some  TO  volumes  of  testimony  and 
supporting  material  aggregating  some  75,000  printed  pages.  For 
each  of  the  last  two  years,  nine  of  the  thirteen  regular  annual  appro- 
priation bills  were  passed  through  the  House  by  the  end  of  June.  In 
each  of  these  years  three  bills  were  delayed  due  to  the  lack  of  legisla- 
tive authorization.  This  year  we  again  hope  to  clear  as  many  of  the 
appropriation  bills  as  possible  through  the  House  by  the  end  of 
June  contingent  upon  the  availability  of  the  necessary  authorizing 
legislation. 

The  Committee  has  now  begun  hearings  on  the  fiscal  1976  budget. 
1  hiring  the  hearings  this  year,  the  Committee  will  explore  certain  pro- 
grams and  situations  which  have  been  the  subject  of  special  studies  by 
the  Committee's  Surveys  and  Investigation  Staff.  The  Committee  has 
over  a  number  of  years  delegated  the  power  to  originate  requests  for 
investigations  to  its  subcommittees,  wTith  the  concurrence  of  the  Chair- 
man and  Kanking  Minority  Member  of  the  full  Committee  required  to 
implement  such  requests.  The  various  subcommittees  utilize  the  capa- 
bility of  the  Surveys  and  Investigations  Staff  to  make  examinations  of 
specific,  subjects. 

As  a  matter  of  information,  the  Surveys  and  Investigations  StalF 
is  headed  by  a  permanent  professional  staff  member  wdio  is  assisted 
by  three  high-caliber  professional  personnel  from  the  Federal  Bureau 
of  Investigation  under  an  arrangement  dating  back  to  1943.  These  men 
are  obtained  from  the  F.B.I,  and  are  placed  directly  on  the  payroll  of 
the  Appropriations  Committee  for  the  duration  of  their  tour.  Each 
serves  two  years  in  an  apprenticeship  capacity  and  then  one  year  as  a 
senior  man.  After  this  three-year  assignment  is  completed,  the  men  re- 
turn to  the  Bureau  to  continue  their  professional  careers. 

When  an  investigation  is  ordered,  experts  in  the  field  are  assembled 
on  an  ad  hoc  basis,  from  outside  or  from  within  the  government,  to 
conduct  the  prescribed  study  and  prepare  a  report  thereon  for  the  use 
of  the  Committee.  The  Committee  delegates  to  this  staff  the  responsi- 
bility for  the  selection  of  individual  experts,  the  criteria  being  com- 
petence and  objectivity. 

Once  a  particular  study  is  completed,  the  investigator  returns  to  his 
parent  organization  which  is  appropriately  reimbursed  for  his  time. 
Since  the  investigators  are  not  permanent  personnel,  the  system  is  elas- 
tic and  economical.  It  is  a  unique  arrangement  insofar  as  we  know,  and 
the  Committee  has  been  well  satisfied  with  the  unbiased,  objective  re- 
ports rendered. 

During  the  93rd  Congress,  the  Investigative  Staff  conducted  95  stud- 
ies concerning  various  agencies*  operations  and  programs.  During 
this  two-year  period.  134  people  on  loan  from  18  agencies  of  the  Gov- 
ernment spent  1.489  man-months  engaged  in  the  conduct  of  studies  and 
the  preparation  of  reports.  This  is  equivalent  to  about  124  man-years. 
Some  of  these  studies,  together  with  studies  initiated  this  year,  will  lie 
an  important  part  of  the  Committee's  activities  this  year  which  should 
be  classified  as  "oversight'"  in  nature.  There  follows  a  listing  of  studies 
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•and  investigations  which  have  been  initiated  as  a  result  of  the  hearings 
last  year  or  in  anticipation  of  hearings  for  fiscal  year  1976  and  repre- 
sent subjects  which  are  candidates  for  "oversight"  attention  this  year. 

A  rt.tt.nl    I  n»-   rati 

Name  of  study 

A  detailed  review  of  the  operations  of  the  Defense 
Investigative  Service  of  the  Department  of  Defense. 

Programs  and  activities  of  the  National  Security  Agency- 
Utilization  of  military  facilities — closure  of  McCoy  Air 
Force  Base. 

Phasedown  of  the  Naval  Air  Station,  Alameda.  Calif 

Phasedown  of  the  Hamilton  Air  Force  Base.  Calif 

Phasedown  of  Ft.  Wainwright,  Fairbanks,  Alaska 

Closure  of  the  Naval  Shipyard,  Hunters  Point.  Calif 

A  review  of  the  navigation  systems  of  the  Federal  Gov- 
ernment. 

Air  traffic  control  systems  in  the  Department  of  Trans- 
portation and  the  Department  of  Defense. 

Proposed  relocation  of  the  Fleet  Missile  Systems  Anal- 
ysis and  Evaluation  Group  of  the  Department  of  the 
Navy. 

A  review  of  the  worldwide  operations  of  the  Defense 
Mapping  Agency. 

Property  maintenance,  repair,  and  minor  constructions 
in  the  Department  of  Defense. 

Budgeting  and  operational  problems  of  the  Federal 
Buildings  Fund  of  the  General  Services  Adminis- 
tration. 
"The  use  of  automatic  data  processing  and  communica- 
tions systems  of  the  Food  and  Drug  Administration. 
Analysis  of  the  operational  costs  at  Howard  University 
compared  with  educational  institutions  of  comparable 
size. 

A  review  of  the  operation  and  activities  of  the  Minority 
Business  Enterprise  program  of  the  Department  of 
Commerce. 

A  review  of  the  Community  Mental  Health  Centers' 
staffing  grants  program  supported  by  the  Department 
of  Health,  Education,  and  Welfare. 

A  review  of  the  Maternal  and  Child  Health  formula 
grant  program  of  the  Department  of  Health,  Educa- 
tion, and  Welfare. 

Analysis  of  the  overhead  reimbursement  policy  of  the 
Department  of  Health,  Education,  and  Welfare. 

Analysis  of  bed  occupancies  and  credits  to  research 
grants  under  the  General  Clinical  Research  Centers 
program  of  the  National  Institutes  of  Health. 

A  review  of  the  evaluation  programs  of  the  Department 
of  Health,  Education,  and  Welfare. 

A  study  of  Federal  programs  which  support  the  purchase 
of  instructional  equipment  by  institutions  of  higher 
learning — Office  of  Education  of  the  Department  of 
Health,  Education,  and  Welfare. 

Analysis  of  the  manpower. management  policy  and  prac- 
tices in  the  Department  of  Health,  Education,  and 
Welfare. 

A  review  and  analysis  of  the  various  problems  associated 
with  the  construction  of  the  Gathright  Lake,  Va., 
flood  control  project,  Army  Corps  of  Engineers. 
A  review  of  the  contracts  and  grants  awarded  by  the  Of- 
fice of  Education  for  the  support  and  improvement  of 
postsecondary  education — Department  of  Health,  Ed- 
ucation, and  Welfare. 


Actual  (or  estimated) 
date  of  completion 
Dec.  16,  1974. 

(Mar.  7.  1975.) 
June  24,  1974. 

Aug.  30,  1974. 
Sept.  24,  1974. 
Sept.  24,  1974. 
Sept.  27,  1974. 
Feb.  12,  1975. 

(Mar.  14,  1975.) 

Nov.  4,  1974. 


(Mar.  7,  1975.) 
(Mar.  19,  1975.) 
Feb.  19,  1975. 

Feb.  10.  1975. 
(Mar.  14,  1975.) 

(Mar.  6,  1975.) 

(Mar.  19,  1975.) 

Feb.  18,  1975. 

(Mar.  24,  1975.) 
Jan.  10,  1975. 

Feb.  25,  1975. 
(Mar.  3,  1975.) 

Feb.  13,  1975. 
Feb.  3,  1975. 
(Mar.  12,  1975.) 
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Same  of  ttudy 

An  analysis  of  headquarters  staffing  and  activities  of  the 
various  components  of  the  Department  of  Defense. 

A  review  and  analysis  of  the  activities  of  the  Consumer 
Product  Safety  Commission. 

An  overall  review  of  the  activities  of  the  Federal  Trade 
Commission. 

A  specific  analysis  of  the  information  systems  and  auto- 
matic data  processing  activities  of  the  Federal  Trade 
Commission. 

An  overall  review  of  the  activities  of  the  Food  and  Drug 
Administration. 

A  general  review  of  the  F-14  aircraft  program  of  the 
Department  of  the  Navy,  including  delivery  schedule, 
advance  payments,  performance  of  the  aircraft,  com- 
parative cost  of  F-14  and  F-15,  and  cost  effectiveness 
compared  with  possible  replacement  aircraft. 

A  review  of  the  activities  of  the  National  Center 
for  Toxicological  Research  of  the  Food  and  Drug 
Administration. 

A  review  of  the  requirements  and  criteria  used  by  the 
military  services  in  providing  graduate  degrees  to  mili- 
tary personnel. 

A  report  to  the  Committee  on  the  factors  which  have 
contributed  to  the  rising  cost  of  electricity  to  the 
American  consumer  and  the  effect  Government  regula- 
tions have  had  on  such  increases. 

A  report  of  the  overseas  deployment  of  tactical  nuclear 
weapons  of  the  Department  of  Defense. 

An  analysis  of  the  Migrant  Health  Program  of  the  De- 
partment of  Health,  Education,  and  Welfare. 

A  review  of  the  airborne  warning  and  control  system 
proposed  by  the  Department  of  the  Air  Force. 

An  inquiry  into  the  operation  of  the  Tokota  Air  Base, 
Japan,  and  the  consolidation  of  military  facilities  in 
the  Kanto  Plain  area. 

The  feasibility  of  adopting  the  unit  replacement  concept 
for  armed  forces  deployed  overseas. 

A  detailed  analysis  of  the  computer  operations  of  the 
Energy  Research  and  Development  Administration. 

A  report  on  the  cost  overruns  associated  with  the  vari- 
ous programs  of  the  Atomic  Energy  Commission  (En- 
ergy  Research   and  Development  Administration). 

A  comprehensive  analysis  of  the  New  Communities  As- 
sistance Program  of  the  Department  of  Housing  and 
Urban  Development. 

Analysis  of  the  scientific  research  programs  of  the  De- 
partment of  Agriculture. 

An  analysis  of  the  regional  and  area  office  structure  of 
the  Department  of  Housing  and  Urban  Development. 

An  analysis  of  the  need  for  and  cost  of  the  Rocket  As- 
sisted Projectile  Program  of  the  Department  of  the 
Army. 

An  analysis  of  the  research  facilities  of  the  Environ- 
mental Protection  Agency. 

A  study  of  the  restructuring  of  the  Naval  Reserve,  to- 
gether with  an  evaluation  of  the  impact  such  restruc- 
turing has  had  upon  the  training,  readiness,  morale, 
etc.  of  the  personnel  involved. 

An  analysis  of  the  need  for  a  proposed  second  source  for 
hull  and  turret  castings  for  the  M48  tank — Depart- 
ment of  the  Army. 


Actual  (or  mtimated) 
date  of  completion 
(Mar.  27,  1975.) 

(Mar.  11,  1975.) 

Feb.  7,  1975. 

Jan.  10,  1975. 

Feb.  26,  1975. 
(May  15,  1975.) 

(Mar.  31,  1975) 
(Apr.  18,  1975.) 
(Mar.  14,  1975.) 

(May  30,  1975.) 
Feb.  20,  1975. 
(Apr.  11,  1975.) 
(June  13,  1975.) 

(May  26, 1975.) 
(Mar.  31,  1975.) 
(Mar.  26,  1975.) 

(Mar.  31,  1975.) 

(Mar.  14, 1975.) 
(Mar.  21,  1975.) 
(Apr.  2,  1975.) 

(Mar.  21,  1975.) 
(Apr.  21,  1975.) 

(Feb.  4,  1975.) 
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Actual   (or  estimated) 
Xame  of  study  date  of  completion 

A  detailed  review  of  the  construction  programs  of  the     (May  1,  1975.  t 

Veterans'  Administration,  including  an  analysis  of  cost 

overruns  associated  therewith. 
A  detailed  analysis  of  the  XJI-1  tank  program  of  the     (July  11,  1975.) 

Department  of  the  Army. 
A  review  of  the  compliance  of  the  Department  of  the     (Mar.  21,  1975.) 

Interior  with  certain  language  contained  in  the  House 

Report  for  fiscal  year  1975  concerning  the  Intermoun- 

tain  Indian  School  and  the  National  Indian  Training 

Center  at  Brigham  City,  Utah. 

"With  respect  to  the  above  listing,  it  is  important  to  note  that  because 
the  studios  originate  with  the  subcommittees,  any  information  devel- 
oped during  the  course  of  an  investigation  is  reported  to  the  chair- 
man of  the  subcommittee  which  requested  such  study  or  examination 
as  well  as  the  Chairman  and  Ranking  Minority  Member  of  the  full 
Committee,  and  such  information  may  be  released  for  publication 
only  when  the  subcommittee  so  determines.  This  procedure  is  in- 
corporated in  Section  8  of  the  Committee's  rules. 

The  above  listing  should  not  be  interpreted  as  the  full  extent  of  the 
Committee's  oversight  activities.  Requests  for  investigations  are  initi- 
ated by  the  various  subcommittees  from  time  to  time  as,  in  the  judg- 
ment of  their  members,  are  needed  to  illuminate  a  particular  appro- 
priations issue  or  problem  related  to  a  department  or  agency  within 
their  respective  jurisdictions.  It  would  not  be  practical  to  undertake 
to  identify  what  further  subjects  will  receive  attention  as  might  be 
generated  by  the  regular  annual  review  process  which  has  just  begun. 

In  addition  to  the  information  made  available  to  the  Committee 
through  its  Surveys  and  Investigations  Staff,  its  "oversight"'  func- 
tions are  buttressed  by  the  continual  utilization  of  the  resources  of  the 
General  Accounting  Office.  The  Committee  receives  a  copy  of  every 
GAO  report  sent  to  the  Congress.  In  recent  years  the  scope  of  the 
GAO  auditing  and  review  capability  has  been  enlarged  to  include 
management  surveys.  For  some  seventeen  years,  the  GAO  has  ex- 
tracted from  these  reports  and  compiled  in  separate  volumes  a  list  of 
so-called  ''significant  Audit  Findings''  for  special  use  by  the  Commit- 
tee staff  in  the  annual  appropriations  hearings.  This  compilation  is 
designed  to  identify  problem  areas  in  an  individual  agency  which 
might  have  applicability  to  other  organizations.  These  findings  relate 
to  matters  which  are  felt  to  require  corrective  action  by  the  Commit- 
tee's efforts,  through  legislation,  or  through  administrative  efforts. 
You  and  your  staff  may  be  interested  in  reviewing  the  latest  191 
page  report  which  was  issued  by  the  General  Accounting  Office  on 
January  2,  1975. 

The  extent  of  use  of  GAO  findings  by  the  Appropriations  Com- 
mittee is  reflected  in  the  number  of  references  to  the  GAO  in  the  an- 
nual appropriation  hearings  on  the  House  and  Senate  in  recent  years 
as  follows : 

Year : 

1968  649 

1969 727 

1970 701 

1971  722 

1972  722 

1973  632 
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Additionally,  the  Committee  frequently  calls  upon  GAO  to  make 
special  studies  and  investigations.  During  FY  1974  GAO  prepared 
10  studies  of  significant  programs  as  requested  by  the  Committee  as 
well  as  .53  staff  studies  on  defense  weapons  systems. 

Pending  for  possible  attention  in  connection  with  the  fiscal  year 
1976  hearings  are  GAO  studies  in  a  number  of  areas.  Selected  examples 
are  as  follows : 

Request  for  views  on  section  1015  of  Public  Law  93-344  with 
regard  to  budget  rescissions  and  deferrals 

Comprehensive  review  of  commissary  operation  and  commissary 
stores  operated  by  the  military  services 

Review  all  funds  obligated  under  the  Presidential  Transition 
Act 

Army's  program  to  modernize  ammunition  plants — Depart- 
ment of  the  Army  (continuing  work) .  B-172707 
International  shipments  of  household  goods — DOD 
Move  policies  related  to  unaccompanied  tours  of  duty 
"Do  It  Yourself"  programs  for  movement  of  household  goods 
Planning  and  budgeting  for  new  test  facilities  at  Arnold  En- 
gineering Development  Center,  Tennessee 

Department  of  Defense  Requirement  for  Helicopters 
Feasibility  of  Consolidating  Chaplain  Schools 
Training  Rates  for  Pilots  and  Navigators 

Legality  of  crediting  reimbursements  to  the  appropriation  year 
collected.  Also,  requests  GAO  in  conjunction  with  OMB  and 
Treasury  Department,  reexamine  the  appropriation  and  appor- 
tionment processes  to  determine  what  legislative  and  regulatory 
changes  may  be  needed  to  eliminate  the  existing  potential  for 
the  M  accounts  and  related  Surplus  Funds  accounts  to  be  used  to 
increase  fund  availability  and  obligational  authority  to  avoid 
possible  overobligation  or  overexpenditure  violations  under  the 
Ant i -Deficiency  Act 

Study  allegations  that  the  Army  Electronic  Command  at  Fort 
Monmouth.  New  Jersev,  has  overobligated  funds  available  to  it 
for  FY  1974  by  some  $56  million 

Review  directed  by  Conference  Report  on  FY  DOD  Appro- 
priations. House  Report  Number  93-1363,  dated  9/18/74— DOD*s 
stockpiling  of  war  materials  or  equipment  for  use  by  foreign 
countries 

Review  of  enlisted  separation  payments  (Directed  by  House 
Report  No.  93-1255,  Fiscal  Year  1975  Department  of  Defense 
Appropriation  Bill) 

Loan  Guarantee  Programs  under  the  Rail  Passenger  Service 
Act  and  the  Regional  Rail  Reorganization  Act 

Cost  comparisons  of  subsidized  housing  provided  under  the 
Section  23.  Section  236,  and  revised  Section  23  programs,  and 
monitor  and  keep  Congress  advised  of  costs  of  the  new  revised 
Section  23  program  as  it  is  being  developed  and  implemented. 
In  1972  GAO  was  reorganized  on  a  functional  basis.  This  was  de- 
signed to  encourage  better  response  to  demands  for  broad-based  and 
program-type  reviews.  It  provided  a  vehicle  for  senior  staff  to  achieve 
a  high  degree  of  specialization  in  the  subject  matters  of  functional 
areas  assigned  to  them. 
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Further,  an  Office  of  Program  and  Budget  Analysis  was  established 
in  1975  in  response  to  the  Budget  and  Impoundment  Control  Act 
of  1974.  The  specific  responsibility  of  this  office,  to  the  Appropriations 
Committee  and  others,  includes  the  following : 

1.  Budget  issue  identification  and  analysis, 

•2.  Macro  analysis  of  major  budget  and  fiscal  issues, 

3.  Budget  related  revenue  analyses, 

4.  Analysis  of  executive  budget  and  supporting  data, 

5.  Coordination  of  GAO-wide  support  of  the  Congressional 
budget  process. 

6.  Impoundment  review  and  reporting. 

7.  Liaison  with  Congressional  Budget  Office  and  the  Con- 
gressional Budget  Committees  and  liaison  with  other  Congres- 
sional Committees. 

The  Committee  is  presently  contemplating  how  best  to  further  and 
better  utilize  the  GAO  in  regard  to  their  traditional  services  and 
expanding  capabilities  but  especially  with  respect  to  securing  rapid 
reports  on  topics  which  relate  to  issues  arising  in  current  appropria- 
tions hearings.  It  is  felt  that  such  a  capability  may  be  especially  fruit- 
ful in  the  defense  field. 

"While  I  realize  that  some  of  my  comments  in  regard  to  the  "over- 
sightv  plans  of  the  Appropriations  Committee  have  been  somewhat 
general,  I  hope  the  specifics  that  I  have  indicated  are  sufficient  to 
illustrate  the  scope  and  thrust  of  the  work  we  undertake  on  an  annual 
basis  and  will  be  useful  to  you  in  developing  the  report  which  must 
be  rendered  in  accordance  with  Clause  2  of  Rule  X.  If  there  is  any 
other  information  which  we  might  provide,  we  want  to  cooperate  to 
the  fullest  extent. 

Best  regards. 
Sincerely, 

George  Mahon,  Chairman. 
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U.S.  House  of  Representatives, 

Committee  on  Armed  Services, 
Washington,  D.C.,   February  28,  1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations, 
U.S.  Ilouse  of  Representatives,  Washington,  D.C. 

Dear  Mr.  Chairman  :  This  will  acknowledge  your  letter  of  Feb- 
ruary 13,  1975  with  reference  to  the  provisions  of  clause  2(c)  of 
Rule  X  of  the  rules  of  the  House  for  the  94th  Congress  requiring 
coordination  of  the  oversight  activities  of  the  standing  Committees 
of  the  House  by  the  Committee  on  Government  Operations.  We  have 
consulted  with  members  of  your  staff  and  submit  the  following  report 
of  oversight  contemplated  by  the  Armed  Services  Committee  during 
the  94th  Congress. 

This  Committee  will  not  utilize  a  special  oversight  subcommittee 
to  carry  out  its  oversight  responsibilities.  Rather,  as  has  been  the  case- 
over  the  years,  the  standing  legislative  subcommittees  and  the  Special 
Subcommittee  on  Intelligence  will  be  responsible  for  the  oversight 
function  in  their  areas  of  jurisdiction. 

The  general  oversight  responsibilities  of  the  House  Armed  Services 
Committee,  as  contained  in  Rule  X  of  the  rules  of  the  House  of 
Representatives,  are  as  follows: 

(1)  Common  defense  generally. 

(2)  The  Department  of  Defense  generally,  including  the  Depart- 
ments of  the  Army,  Navy,  and  Air  Force  generally. 

(3)  Ammunition  depots:  forts;  arsenals;  Army,  Navy,  and  Air 
Force  reservations  and  establishments. 

(4)  Conservation,  development,  and  use  of  naval  petroleum  and  of 
shale  reserves. 

(5)  Pay.  promotion,  retirement,  and  other  benefits  and  privileges 
of  members  of  the  armed  forces. 

(6)  Scientific  research  and  development  in  support  of  the  armed 
services. 

(7)  Selective  service. 

(8)  Size  and  composition  of  the  Army,  Navy,  and  Air  Force. 

(9)  Soldiers'  and  sailors'  homes. 

(10)  Strategic  and  critical  materials  necessary  for  the  common- 
defense. 

Note. — Under  Rule  X  3(a)  the  Committee  has  a  special  oversight 
function  with  respect  to  international  arms  control  and  disarmament 
and  military  dependents  education. 
(21) 
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In  response  to  the  House  rules  and  your  specific  request,  the  follow- 
ill*;  areas  and  subjects  have  been  designated  for  special  oversight  at- 
tention during  the  94th  Congress : 

(1)  Intelligence 

To  cover  the  Central  Intelligence  Agency,  the  National  Security 
Agency  and  Department  of  Defense  and  service  intelligence  com- 
ponents, including  intelligence  activities  in  the  United  States  and 
abroad  and  the  collection,  analysis  and  dissemination  of  intelligence 
information,  intelligence  estimates  and  covert  activities.  Also,  in- 
cluded are  those  areas  where  such  departments  or  agencies  participate 
in  intelligence  matters  outside  their  respective  immediate  chains  of 
command.  The  overview  will  also  include  personnel  policies  and  prac- 
tices, relevant  procurement  policies  and  practices  and  budget  matters. 

(2)  Energy 

The  exploration,  development  and  any  limited  production  in  the 
Naval  Petroleum  and  Oil  Shale  Reserves  with  particular  reference 
to  the  maintenance  of  a  defense  petroleum  asset. 

(3)  Foreign  Military  Sales  and  Their  Impact  on  Readiness  of 
U.S.  Forces 

(4)  U.S.  Training  of  Forei£rn  Military  Personnel 

(.">)   Cost-Effectiveness  Studies  of  tlie  Military  Departments 

(6)  Foreign  Investment  in  U.S.  Defense-Oriented  Corporations 

(7)  Department  of  Defense  "World-Wide  Command  and  Con- 
trol System 

(8)  Thefts  and  Losses  of  Military  Weapons 
(0)  Military  Commissary  Stores 

(10)  Strategic  and  Critical  Materials  in  the  National  Stockpile 

(11)  Military  Real  Estate  Acquisitions  and  Disposals 

(12)  The  Naval  Base  at  Diego  Garcia 

(13)  T1  e  Airborne  Command  Post  Program 

(14)  The  Manned  Bomber  in  the  Strategic  Arsenal 
(lft)   The  Planned  Production  Contract  for  AW  ACS 

(16)  Naval  Vessels  and  Shipyards 

Special  attention  will  be  given  to  an  examination  of  the  procure- 
ment and  construction  of  naval  vessels  and  maintenance  and  produc- 
tion problems  in  public  and  private  shipyards. 

(17)  Anti-Submarine  Warfare 

This  overview  will  include  procurement  and  construction  of  anti- 
submarine warfare  systems. 

(15)  Transfer  and  Disposal  of  Naval  Vessels 
(10)   Sealift  in  our  National  Security 

(20)  Department  of  Defense  Personnel  (Military  and  Civilian) 
Particular  attention  will  be  focused  on  the  all-volunteer  force  con- 
cept. tli°  total  force  concept,  retention  of  military  personnel,  military 
medical  care  delivery  and  the  CHAMPUS  program,  dependent  poli- 
cies in  the  U.S.  and  overseas  and  the  Uniform  Code  of  Military 
Justice. 

(21^  Military  Compensation.  Promotion  and  Retirement 
Attention  will  be  focused  on  proposals  which  involve  revision  of 
the  total  spectrum  of  laws  relating  to  officer  selection,  promotion  and 
separation  procedures,  as  well  as  modernization  of  the  military  retire- 
ment >vstem.  Also,  there  will   be  a  reexamination  of  the  Survivor 
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Benefit  Plan  for  military  personnel.  In  addition,  there  will  be  a  major 
review  of  the  total  range  of  military  pay  and  allowances,  including 
the  implementation  of  special  pay  legislation  for  professionals  in  the 
health  care  fields,  personnel  in  military  flight  status  and  other  areas 
of  special  pav  categories. 

(22)  the  Armv  Tank  Program 

(23)  U.S.  Military  Commitment  to  NATO 

(24)  The  Soviet  Threat 

(25)  The  Military  Situation  in  the  Middle  East 

(26)  Arms  Control  and  Disarmament 

( 27 )  The  Selective  Service  System 

(28)  Research  and  Development 

General  legislative  oversight  incorporating  all  of  the  research,  de- 
velopment, test  and  evaluation  programs  of  the  individual  service  de- 
partments and  defense  agencies,  with  special  emphasis  on  those  pro- 
grams which  are  being  considered  for  ultimate  procurement,  such  as 
the  AWACS;  15—1 ;  the  XM-1  tank  program;  various  missile  pro- 
grams and  support  systems;  the  TRIDENT  program;  military  air- 
lift :  independent  research  and  development.  The  Committee  also  in- 
tends to  review  the  following  subject  matter  in  connection  with  its 
RDT&E  oversight  responsibilities:  military  requirements  and  mili- 
tary support  technology  for  chemical  weapons ;  development  of  greater 
standardization  and  commonality  in  military  weapons  systems;  the 
implications  of  detente  on  RDT&E  effort ;  the  Strategic  Arms  Limita- 
tion Treaty;  and  various  other  foreign  policy  implications  in  the 
RDT&E  programs  of  the  military  departments. 

I  believe  it  is  evident,  Mr.  Chairman,  that  the  foregoing  informa- 
tion is  not  restrictive  in  any  way  since  experience  indicates  that  future 
circumstances  may  require  additional  oversight  emphasis  in  areas  not 
previously  marked  for  special  attention.  In  such  instances  it  may  well 
be  necessary  to  divert  the  Committee  resources  in  new  directions  at 
the  expense  of  an  area  or  areas  previously  marked  for  special  atten- 
tion. Also,  in  submitting  the  information  above  concerning  intelligence 
oversight,  we  are  mindful  of  the  special  requirements  placed  upon  the 
Select  Committee  on  Intelligence  established  by  H.  Res.  138.  In  addi- 
tion, it  should  be  pointed  out  that  the  Committee  conducts  continuous 
oversight  activities  through  correspondence  and  other  communications 
into  all  facets  of  the  provisions  of  Titles  10  (Armed  Forces).  37  (Pay 
and  Allowances),  50  and  50  Appendix  (War  and  National  Defense), 
United  States  Code  by  virtue  of  the  unique  wav  such  statutes  cut 
across  the  daily  operations  and  activities  in  the  Department  of  De- 
fense and  in  some  instances,  the  Central  Intelligence  Agency.  Finally, 
in  carrying  out  its  oversight  responsibility  the  Committee  on  Govern- 
ment Operations  may  care  to  have  reference  to  the  Report  of  Activi- 
ties of  the  Committee  on  Armed  Services,  93rd  Congress,  House  Report 
93-1662  of  January  2, 1975. 

With  best  regards,  I  am, 
Sincerely, 

Melvin  Price,  Chairman. 
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U.S.  House  of  Representatives, 
Committee  on  Banking,  Currency  and  Housing, 

Washington,  D.C.,  February  28, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Government  Operations  Committee,  House  of  Representa- 
tives, Washington,  D.C. 
Dear  Mr.  Chairman  :  In  reference  to  your  recent  letter  and  discus- 
sions between  members  of  your  staff  and  the  staff  of  the  House  Com- 
mittee on  Banking,  Currency  and  Housing,  the  following  information 
is  submitted  to  you  concerning  the  oversight  activities  of  our 
Committee. 

The  House  Committee  on  Banking,  Currency  and  Housing  has  estab- 
lished a  Subcommittee  on  General  Oversight  and  Renegotiation.  The 
jurisdiction  of  this  Subcommittee,  as  set  forth  in  the  Committee  Rules, 
states  as  follows: 

The  General  Oversight  and  Renegotiation  Subcommittee  shall 
assist  the  House  Committee  on  Banking,  Currency  and  Housing 
in  appraising  the  administration  of  the  laws  and  regulations  under 
the  jurisdiction  of  the  Committee  and  present  such  recommenda- 
tions as  deemed  necessary  to  the  appropriate  subcommittee (s) 
of  the  Committee. 

Further,  the  Subcommittee  shall  exercise  continuing  oversight 
of  the  execution  by  the  administrative  agencies  concerning  any  of 
the  laws  the  subject  matter  of  which  reside  within  the  jurisdiction 
of  the  Committee  and  shall  study  all  pertinent  reports,  documents 
and  data  pertinent  to  the  jurisdiction  of  the  Committee  and  make 
the  necessary  recommendations  or  reports  thereon  to  the  appropri- 
ate subcommittee (s)  of  the  Committee. 
Further,  the  Subcommittee  shall  have  full  jurisdiction  over  the 
Renegotiation  Act  of  1951,  as  amended. 

Further,  as  stated  in  Rule  IX  of  the  Rules  of  the  Committee  on 
Banking,  Currency  and  Housing,  the  Oversight  Subcommittee  is 
charged  with  responsibilities  as  set  forth  in  Attachment  1.  It  is  under- 
stood, as  set  forth  in  this  rule,  that  the  Oversight  Subcommittee  of  the 
House  Committee  on  Banking,  Currency  and  Housing  shall  not  in- 
fringe upon  the  jurisdiction  of  another  subcommittee  of  the  full  Com- 
mittee without  proper  consultation.  The  Oversight  Subcommittee, 
therefore,  is  directed  to  coordinate  and  cooperate  with  the  legislative 
subcommittees  of  the  full  Committee.  The  Oversight  Subcommittee  is 
precluded  from  investigating  any  subject  matter  falling  within  the 
jurisdiction  of  a  legislative  subcommittee  of  the  Committee  unless  the 
legislative"  subcommittee  is  in  accord  with  the  undertaking. 
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The  Chairman  of  the  Committee  and  the  Committee  is  to  be  in- 
formed of  any  proposed  oversight  undertakings  by  any  subcommittee. 
If  any  question  arises  concerning  the  jurisdiction  of  the  Oversight 
Subcommittee  and  a  legislative  subcommittee,  the  Chairman  of  the 
Committee  is  charged  with  the  responsibility  of  arbitrating  the  dif- 
ferences. If  necessary,  the  matter  can  be  appealed  to  the  Democratic 
Caucus  and,  if  necessary,  the  full  Committee. 

Some  of  the  oversight  and  investigative  activities  of  the  various 
subcommittees  of  the  House  Committee  on  Banking,  Currency  and 
Housing  that  will  be  undertaken  this  year  are  set  forth,  by  subcom- 
mittee, in  Attachment  2.  This  list  of  proposed  oversight  and  investi- 
gative activities  of  the  various  subcommittees  of  the  full  Committee 
may  not  be  all-inclusive.  Instances  may  arise  requesting  immediate 
attention  by  the  committee  or  one  of  its  subcommittees  which  have 
not  been  anticipated.  Further,  the  amount  of  time  devoted  to  a  given 
investigation  may  exceed  the  initial  estimated  time  and,  therefore,  it 
may  not  be  possible  to  carry  out  oversight  in  all  of  the  areas  detailed 
above. 

Mr.  Chairman,  I  believe  the  above  statements  answers  m  detail 
your  questions  concerning  this  Committee's  oversight  structure  and 
functioning. 

Sincerely, 

Hent?y  S.  Keuss,  Chairman. 

[Attachment  1] 

Rule  No.  IX 

OVERSIGHT  subcommittee 

^a)  In  order  to  assist  the  House  in: 

(1)  its  analysis,  appraisal,  and  evaluation  of  (A)  the  applica- 
tion, administration,  execution,  and  effectiveness  of  the  laws 
enacted  by  the  Congress,  or  (B)  conditions  and  circumstances 
which  may  indicate  the  necessity  or  desirability  of  enacting  new 
or  additional  legislation,  and 

(2)  its  formulation,  consideration,  and  enactment  of  such  mod- 
ifications or  changes  in  those  laws,  and  of  such  additional  legis- 
lation, as  may  be  necessary  or  appropriate,  there  shall  be  estab- 
lished in  conformity  with  Rule  XV  an  Oversight  Subcommittee. 

(b)  The  Oversight  Subcommittee  shall  review  and  study,  on  a  con- 
tinuing basis,  the  application,  administration,  execution,  and  effective- 
ness of  those  laws,  or  parts  of  laws,  the  subject  matter  of  which  is 
within  the  jurisdiction  of  the  committee,  and  the  organization  and 
operation  of  the  Federal  agencies  and  entities  having  responsibilities 
in  or  for  the  administration  and  execution  thereof,  in  order  to  deter- 
mine whether  such  laws  and  the  programs  thereunder  are  being  imple- 
mented and  carried  out  in  accordance  with  the  intent  of  the  Congress 
and  whether  such  programs  should  be  continued,  curtailed,  or  elimi- 
nated. In  addition,  the  Oversight  Subcommittee  shall  review  and 
study  any  conditions  or  circumstances  which  may  indicate  the  necessity 
or  desirability  of  enacting  new  or  additional  legislation  within  the 
jursdiction  of  the  committee  (whether  or  not  any  bill  or  resolution 
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has  been  introduced  with  respect  thereto),  and  shall  on  a  continuing 
basis  undertake  futures  research  and  forecasting  on  matters  within 
the  jurisdiction  of  the  committee.  The  Oversight  Subcommittee  shall 
in  no  way  limit  the  responsibility  of  the  subcommittees  from  carrying 
out  their  oversight  responsibilities. 

(c)  The  Oversight  Subcommittee  shall  review  and  study  on  a  con- 
tinuing basis  the  impact  or  probable  impact  of  tax  policies  affecting 
subjects  within  the  jurisdiction  of  the  committee. 

DOMESTIC    MONETARY    POLICY    SUBCOMMITTEE 

This  subcommittee  in  the  immediate  future  will  begin  hearings  on 
legislation  which  would  require  interest  to  be  paid  on  funds  owned  by 
the  Federal  Government  residing  in  non-interest  bearing  accounts, 
primarily  in  commercial  banks.  This  has  been  a  matter  which  has  been 
under  investigation  for  a  number  of  years,  and  finally  the  Treasury 
Department  has  agreed  that  such  funds  should  bear  interest.  Estimates 
indicate  the  Federal  Government  could  realize  upwards  of  $400  million 
a  year  from  such  legislation. 

Investigative  undertakings  of  the  Subcommittee  this  year  will  in- 
clude inquiry  into: 

The  necessity  and  desirability  for  monetary  policy  to  be  made  an  in- 
tegral part  of  the  overall  economic  policy-making  process  now  carried 
on  by  the  Executive  Branch  and  the  Congress,  thus  making  monetary 
policy  more  directly  accountable  to  the  political  process. 

The  relationship  between  the  monetary  policy  responsibilities  of  the 
Federal  Reserve  and  its  bank  regulatory  responsibilities,  including  the 
questions  of  whether  a  serious  conflict  of  interest  exists  in  having  the 
same  government  agency  carrying  out  at  the  same  time  both  monetary 
and  regulatory  responsibilities;  and  the  extent  to  which  the  Federal 
Reserve  uses  its  regulatory  power  to  achieve  its  monetary  policy  goals. 

To  what  extent  does  the  Federal  Reserve  use  its  authority  to  allocate 
credit  on  an  ad  hoc  basis  while  publicly  opposing  Congressional  at- 
tempts to  establish  a  fair,  open  and  rational  system  of  credit  allocation 
with  specifically  established  criteria  based  on  Congress'  perception  of 
the  public  interest. 

The  effect  of  monetary  policy  and  other  financial  actions  on  interest 
rates,  allocation  of  credit,  and  the  structure  and  functioning  of  do- 
mestic and  foreign  financial  institutions,  including  such  possible  effects 
as  creating  increased  concentration,  a  lessening  of  competition,  and 
restricting  adequate  access  to  credit  to  various  elements  in  the  economy. 

HOUSING  AND  COMMUNITY  DEVELOPMENT  SUBCOMMITTEE 

Investigative  oversight  activities  of  the  Subcommittee  on  Housing 
and  Community  Development  this  year  will  include : 

Oversight  hearings  on  the  New  Communities  Program  and  on 
HUD  administration  and  suspension  of  new  communities  program. 

Mortgage  credit  hearings  and  studies  to  provide  a  comprehensive 
review  of  the  role  and  operations  of  the  FHA,  FNMA,  GNMA, 
FHLMC,  Farmers'  Home  Administration,  the  private  mortgage  in- 
surance industry,  and  State  and  local  credit  programs. 
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Section  8  Housing  Subsidy  and  Community  Development  Block 
Grant  Programs,  oversight  of  HUD  implementation  of  programs 
and  evaluation  of  performance  under  programs. 

701  Comprehensive  Planning  Program;  A-95  Review  Process, 
examination  of  program  benefits  and  administration  of  programs. 

Preservation  of  the  existing  housing  stock,  study  of  the  progress 
and  procedures  for  the  preservation  of  older  housing  and  the  con- 
tribution of  the  Section  8  and  Community  Development  Programs. 

ECONOMIC    STABILIZATION'     SUBCOMMITTEE 

The  Economic  Stabilization  Subcommittee  will  devote  a  great  deal 
of  its  time  to  oversight  inquiries  into  the  operations  of  the  Defense 
Production  Act.  This  inquiry  will  explore  the  procedures  and  proc- 
esses whereby  under  the  Defense  Production  Act,  loans  and  loan 
guarantees  can  be  granted  to  businesses  for  the  development  of  tech- 
nological processes  and  for  the  exploration,  development  and  mining 
of  strategic  metals  and  materials.  This,  obviously,  could  have  some 
positive  implications  for  energy  resource  development.  In  addition, 
oversight  investigations  in  this  area  will  include  the  operations  of 
the  1  >efense  Production  Act,  regarding  stockpiling  provisions  for  na- 
tional defense  purposes — both  the  acquisition  and  sale  thereof.  Ob- 
viously, too,  the  stockpile  provisions  of  the  Defense  Production  Act 
could  be  used  for  stockpiling  of  energy  resources. 

FINANCIAL     INSTITUTIONS     SUPERVISION,     REGULATION     AND     INSURANCE 
SUBCOMMITTEE 

This  Subcommittee  intends  to  conduct  investigative  oversight  hear- 
ings into  the  reasons  for  the  recent  bank  failures,  which  hopefully 
will  ultimately  provide  the  Committee  with  the  necessary  ingredients 
needed  to  revive  and  revamp  the  financial  institutions  federal  regula- 
tory structure. 

This  Subcommittee  will,  in  addition,  continue  to  investigate  red- 
lining, whereby  access  to  credit  sources  in  central  cities  in  viable 
neighborhoods  is  denied  credit  due  to  reasons  which  are  not  economic. 
Also,  the  Subcommittee  will  investigate  the  question  of  conversion  of 
mutual  savings  and  loan  associations  into  stock  associations  and  re- 
view existing  cease  and  desist  authority  for  bank  and  savings  and 
loan  holding  companies. 

HISTORIC    PRESERVATION    AND   COINAGE    SUBCOMMITTEE 

The  Subcommittee  on  Historic  Preservation  and  Coinage,  under 
the  Chairmanship  of  the  Honorable  Robert  G.  Stephens,  Jr.,  will  look 
into  the  coinage  programs  of  the  United  States  Government  as  they 
are  being  developed  as  part  of  our  Bicentennial  celebration. 

In  addition,  this  Subcommittee,  again  in  anticipation  of  the  Bicen- 
tennial celebrations  will  conduct  a  complete  review  of  Title  I  of  the 
National  Housing  Act  as  it  relates  to  the  funding  of  studies,  acquisi- 
tions and/or  preservation  of  historic  structures  in  the  United  States. 
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INTERNATIONAL    DEVELOPMENT    INSTITUTIONS    AND    FINANCE 
SUBCOMMITTEE 

Investigative  and  oversight  activities  of  the  Subcommittee  will  com- 
prise a  general  review  of  U.S.  participation  in  the  various  inter- 
national development  institutions  with  a  view  to  determining  what 
the  continuing  role  of  the  U.S.  as  a  participant  should  be  in  the 
institutions.  It  has  been  some  years  since  this  subcommittee  conducted 
a  general  review  of  the  internal  operations  and  lending  policies  of 
these  institutions  which  will  be  done  this  year. 

GENERAL  OVERSIGHT  AND  RENEGOTIATION   SUBCOMMITTEE 

General  oversight  subjects  which  may  be  undertaken  by  this  sub- 
committee, in  consultation  and  accord  with  the  substantive  legislative 
subcommittees  of  the  full  committee,  possibly  would  include  bank 
closing.  Federal  Reserve  System  building  programs,  government  tax 
and  loan  accounts,  Federal  crime  insurance,  military  credit  unions. 
Section  312  rehabilitation  laws  and  laws  of  the  several  states  affecting 
condominium  purchases. 

CONSUMER    AFFAIRS    SUBCOMMITTEE 

Investigative  activities  of  this  Subcommittee  will  include  a  review 
of  the  effectiveness  of  the  existing  truth-in-lending  legislation  and  in- 
vestigation into  the  practices  of  credit  collecting  agencies  and  some 
oversight  hearings  into  specific  consumer  price  increases. 

INTERNATIONAL   TRADE,   INVESTMENT   AND   MONETARY  POLICY 
SUBCOMMITTEE 

The  Subcommittee  on  International  Trade,  Investment  and  Mone- 
tary Policy,  under  the  Chairmanship  of  the  Honorable  Thomas  M. 
Rees,  will  review  the  structure,  function  and  activity  of  the  Export- 
Import  Bank,  especially  as  regards  Ex-Im  lending  and  guarantee 
programs  and  projects  to  Communist  countries. 

In  addition,  the  jurisdiction  of  this  subcommittee  extends  to  IMF 
U.S.  policy  regarding  investments  abroad,  especially  by  multinational 
corporations  and  foreign  investment  in  the  United  States. 

In  addition,  the  Subcommittee  will  begin  an  overall  investigation  of 
operations  and  in  this  regard  legislation  will  probably  be  transmitted 
to  the  Hill  which  would  seek  to  create  an  IMF  special  oil  facility.  If  so, 
this  legislation,  which  would  call  for  the  participation  of  perhaps  up 
to  $25  billion  by  the  United  States  and/or  other  industrial  nations, 
would  be  considered  by  this  Subcommittee. 
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U.S.  House  of  Representatives, 

Committee  on  the  Budget, 
Washington,  D.C.,  March  3, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,   U.S.  House  of 
Representatives,  Washington,  D.C. 

Dear  Mr.  Chairman  :  The  purpose  of  this  letter  is  to  inform  you  of 
the  Budget  Committee's  oversight  responsibilities  in  general,  the 
specific  areas  of  oversight  expected  to  be  undertaken  this  year,  and  the 
structure  to  be  utilized  to  carry  out  oversight  activities. 

As  you  know,  the  Committee  on  the  Budget  only  recently  organized 
itself  for  the  91th  Congress,  and  much  of  the  Committee's  time  has 
been  devoted  to  dealing  with  the  extent  to  which  the  new  budget 
process  called  for  in  the  Congressional  Budget  and  Impoundment 
Control  Act  of  1974  (Public  Law  93-344)  will  be  implemented  this 
year.  Consequently,  since  the  Committee  has  made  no  formal  deci- 
sions on  specific  oversight  activities  to  be  conducted  this  year,  the 
following  should  be  regarded  as  tentative  in  nature.  We  will,  of  course, 
inform  you  as  soon  as  possible  of  specific  oversight  activities  decided 
upon  by  the  Committee. 

GENERAL   OVERSIGHT   RESPONSIBILITIES 

The  Budget  Committee  is  responsible  for : 

(1)  making  continuing  studies  of  the  effect  on  budget  outlays  of 
relevant  existing  and  proposed  legislation  and  reporting  the  results  of 
such  studies  to  the  House  on  a  recurring  basis ; 

(2)  requesting  and  evaluating  continuing  studies  of  tax  expendi- 
tures, devising  methods  of  coordinating  tax  expenditures,  policies, 
and  programs  with  direct  budget  outlays,  and  reporting  the  results  of 
such  studies  to  the  House  on  a  recurring  basis ; 

(3)  reviewing,  on  a  continuing  basis,  the  conduct  by  the  Congres- 
sional Budget  Office  of  its  functions  and  duties.  (P.L.  93-344,  section 
101.) 

In  addition,  P.L.  93-344  directs  the  Budget  Committee  to  undertake 
a  series  of  studies  in  specific  areas : 

(1)  not  later  than  the  close  of  the  95th  Congress,  the  Budget  Com- 
mittee is  required  to  report  to  the  House  on  procedures  used  under 
section  301  (b)  concerning  the  enrollment  of  appropriation  and  entitle- 
ment legislation  (section  301  (b) ) ; 

(2)  on  a  continuing  basis,  the  Budget  Committee  is  required  to  study 
provisions  of  law  which  exempt  certain  Federal  agencies  from  inclu- 
sion in  the  budget  (section  606) ;  and 
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(3)  on  a  continuing  basis,  the  Budget  Committee  is  required  to  study 
proposals  designed  to  improve  and  facilitate  methods  of  congressional 
budget-making  (section  703). 

SPECIFIC   AREAS    OF    OVERSIGHT    TO   BE    UNDERTAKEN    IN    1975 

The  Budget  Committee  has  made  no  formal  decisions  concerning 
which  of  the  studies  enumerated  above  should  be  initiated  this  vear. 
All  involve  important  matters  which  must  receive  continuing  atten- 
tion and  analysis. 

Preliminary  work  has  begun  in  one  area :  a  study  of  tax  expendi- 
tures. The  Committee  recently  asked  the  Joint  Committee  on  Internal 
Revenue  Taxation  to  analyze  the  estimates  of  tax  expenditures  made 
by  the  President  in  the  FY  1976  budget,  and  heard  testimony  on  tax 
expenditures  on  March  3  from  Professor  Stanley  Surrey  of  Harvard 
University.  A  substantial  amount  of  the  Committee's  oversight  activ- 
ity this  year  may  involve  the  tax  expenditures  area. 

The  Committee  will  certainly  give  special  attention  this  year  to 
the  initial  activities  of  the  Congressional  Budget  Office.  This  Office, 
established  on  February  27  upon  appointment  of  a  Director  by  the 
Speaker  of  the  House  and  the  President  pro  tempore  of  the  Senate,  will 
be  organizing  and  staffing  during  the  next  few  months  so  that  it  can 
provide  the  House  and  Senate  Budget  Committees  with  assistance 
during  the  latter  part  of  1975  and,  more  importantly,  in  1976  when 
the  new  congressional  budget  process  is  fully  implemented.  Our  Com- 
mittee will  work  closely  with  the  new  Office  to  assure  that  it  is  or- 
ganized and  staffed  in  a  manner  designed  to  provide  the  highest  quality 
of  service  to  the  Budget  Committees,  other  committees  of  the  House 
and  Senate,  and  Members  of  Congress. 

STRICTURE  TO  BE  USED  TO   CARRY  OUT  OVERSIGHT  RESPONSIBILITIES 

The  Committee  has  made  no  formal  decision  on  the  structure  to  be 
used  to  carry  out  its  oversight  functions.  I  feel  that  a  series  of  task 
forces  should  be  established  to  undertake  the  various  studies  enumer- 
ated above;  and  perhaps  a  special  oversight  task  force  should  be 
established  on  the  Congressional  Budget  Office.  I  plan  to  raise  this 
matter  with  the  Committee  at  an  early  date,  and  will  report  its  deci- 
sions to  you  as  soon  as  possible. 

I  hope  this  information  is  helpful.  Please  let  me  know  if  I  can  be 
of  any  further  assistance. 
Sincerely  yours, 

Brock  Adams,  Chairman. 
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COMMITTEE  ON  THE  DISTRICT  OF  COLUMBIA 


U.S.  House  or  Representatives, 
Committee  on  the  District  of  Columbia, 

Washington,  D.C.,  February  21, 1975. 
Hon.  Jack  Brooks, 

Chairman  of  the  House  Committee  on  Government  Operations,  2157 
It  ay  burn  House  Office  Building,  Washington,  D.C. 
Dear  Mr.  Chairman:  This  comes  in  response  to  your  letter  of 
February  13,  1975,  in  which  you  requested  a  relatively  concise  listing 
of  oversight  activities  contemplated  by  the  District  Committee  during 
the  94th  Congress.  While  the  Committee  has  yet  to  determine  how  the 
oversight  responsibilities  will  be  divided  between  the  full  Committee 
and  six  (6)  subcommittees,  we  are  certain  that  the  following  areas  will 
bo  subject  to  investigative  oversight  by  the  Committee : 

1.  The  operation  of  District  Government  and  the  transition  to  home 
rule. 

2.  St.  Elizabeth's  Hospital. 

3.  The  Bicentennial  organization  and  activities  in  the  National 
Capital  Region. 

4.  The  taxicab  industry  in  the  District  of  Columbia. 

5.  National  capital  regional  issues,  such  as  aspects  of  the  Potomac 
River  and  the  water  and  sewage  system  of  the  District. 

6.  Social  Service  delivery  systems. 

7.  Implementation  of  the  1973  Governmental  Reorganization  plan 
with  a  view  to  its  economy  and  efficiency. 

8.  Operation  of  the  National  Capital  Service  Area. 

9.  METRO  with  respect  to  its  financial  affairs  and  related  matters. 

10.  The  administration  of  justice  in  the  District  of  Columbia,  in- 
cluding an  investigation  of  the  "preventive  detention"  issue. 

11.  Economic  development  and  housing  in  the  city. 

The  Chief  Counsel  of  the  Committee,  Mr.  Robert  B.  Washington, 
Jr.,  is  the  staff  person  designated  for  contact  by  members  of  your  Com- 
mittee Staff.  I,  of  course,  will  be  happy  to  discuss  any  of  these  items 
with  you,  if  you  feel  it  would  be  helpful. 
Sincerely, 

Charles  C.  Diggs,  Jr., 
Chairman,  Committee  on  the  District  of  Columbia. 
(31) 
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COMMITTEE  ON  EDUCATION  AND  LABOR 

House  of  Representatives, 
Committee  on  Education  and  Labor, 

Washington,  D.C.,  March  3, 1975. 
Hon.  Jack  Brooks, 

Chairman*  Committee  on  Government  Operations,  U.S.  House  of  Rep- 
resentatives, Washington,  D.C. 

Dear  Mr.  Chairman:  Pursuant  to  your  letter  of  February  13,  rep- 
resentatives of  this  Committee  conferred  with  your  staff  on  February 
20  on  our  Committee's  oversight  activities  for  the  coming  Congress. 
This  letter  results  from  that  meeting  and  is  an  attempt  to  assist  you  to 
carry  out  your  responsibilities  under  Clause  2(c)  of  title  X  of  the 
Rules  of  the  House. 

At  our  organizational  meeting,  the  Committee  on  Education  and 
Labor  elected  to  continue  to  impose  oversight  responsibility  on  each 
of  our  legislative  subcommittees  rather  than  establish  a  separate  sub- 
con  imittee  which  would  attempt  to  do  oversight  in  the  many  areas 
over  which  we  haA-e  jurisdiction.  The  balance  of  this  letter  will  indi- 
cate what  the  general  jurisdictional  responsibilities  for  each  subcom- 
mittee are  and,  as  far  as  possible,  lay  out  the  present  plans  for  specific 
oversight  activities. 

These  plans  are  necessarily  tentative.  Nor  should  they  be  regarded 
as  exhaustive.  Our  Committee  has  always  felt  obliged  to  respond  to 
crises  or  serious  problems  as  they  arise.  For  example,  the  Special  Sub- 
committee on  Labor  in  the  93d  Congress  watched  very  carefully  the 
development  of  negotiations  and  the  strike  actions  at  the  Brookside 
Mine,  as  it  had  the  West  Coast  dock  strike  in  the  92d  Congress.  Al- 
ready this  year  our  Subcommittee  on  Labor  Standards  has  sent  a  staff 
investigator  to  the  State  of  Washington  to  investigate  the  impact  of 
the  Fair  Labor  Standards  Act  Amendments  of  1974  on  some  of  the 
federal  employees  at  the  Puget  Sound  Naval  Shipyard.  All  of  our 
subcommittees  will  continue  to  be  prepared  to  respond  to  such  problem 
situations  as  the  need  arises.  The  need  for  this  sort  of  response,  the 
development  of  new  and  presently  unforeseen  problems  requiring  at- 
tention, and  our  legislative  duties  may  force  us  to  curtail  some  of  the 
projected  activities.  We  will  continue,  however,  as  we  have  in  years 
past  to  do  as  much  and  as  effective  oversight  as  is  possible. 

Both  your  letter  and  your  staff  emphasized  "concise  listings"  and 
"brevity".  I  hope  the  following  meets  your  needs  in  this  respect  with 
fairness  to  our  subcommittees. 

subcommittee  on  elementary,  secondary  and  vocational  education 

General 

This  Subcommittee  has  jurisdiction  over  matters  affecting  education 
through  the  high  school  level  and  vocational  education. 

(32) 
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Specific 

(1)  The  Subcommittee  will  give  careful  scrutiny  to  the  implementa- 
tion of  the  Education  Amendments  of  1974.  Implementing  regulations 
are  being  developed  and  submitted  to  the  Congress  for  review  and 
oversight  has  already  begun. 

These  amendments  affect  the  Elementary  and  Secondary  Education 
Act,  the  Adult  Education  Act,  Indian  education,  impact  aid,  educa- 
tion of  the  handicapped  and  other  related  laws. 

(2)  Hearings  both  in  Washington  and  in  the  field  are  planned  to 
look  into  Indian  education. 

(3)  There  will  be  a  continuation  of  oversight  hearings  and  investiga- 
tion into  the  administration  and  effectiveness  of  the  Vocational  Edu- 
cation Act  of  1963. 

Legislation  is  expected  to  result  from  these  activities. 

SUBCOMMITTEE   on   labor-management  relations 

General 

This  Subcommittee  has  jurisdiction  over  relationships  between  em- 
ployers and  employees  and  the  federal  laws,  statutes  and  regulations 
relating  to  the  collective  bargaining  process. 

Specific 

(1)  The  Subcommittee  will  continue  oversight  of  the  administra- 
tion of  the  Service  Contract  Act  of  1965  with  a  view  to  bringing  to 
light  certain  questions  of  interpretation  and  policy  that  have  arisen. 

1 2)  Oversight  will  continue  into  the  administration  of  the  National 
Labor  Relations  Act. 

(3)  The  appropriateness  and  effectiveness  of  the  Consumer  Price 
Index  and  its  administration  will  be  studied. 

(4)  The  Labor  Department's  administration  of  the  Walsh-Healey 
Act  will  be  the  subject  of  oversight  activity. 

SUBCOMMITTEE    ON    LABOR  STANDARDS 

Genera? 

As  its  name  implies,  this  Subcommittee  has  jurisdiction  over  wage 
and  hour  standards  including  welfare  and  pension  plans  and  coal 
mine  health  and  safety. 

Specific 

(1)  The  Subcommittee  will  study  the  impact  and  implementation 
of  the  Fair  Labor  Standards  Amendments  of  1974,  more  precisely  the 
coverage  of  domestic  service  employees,  public  employees  (federal, 
State  and  local)  and  the  child  labor  provisions,  particularly  in 
agriculture. 

(2)  The  Subcommittee  will  continue  its  study  of  the  impact  of  im- 
ports and  exports  on  American  employment  and  on  the  wage  and 
hour  standards  affecting  American  business.  Particular  emphasis  will 
be  given  to  the  automobile  industry. 

(3)  The  administration  of  the  Black  Lung  Benefits  Program  (title 
IV  of  the  Federal  Coal  Mine  Health  and  Safety  Act  of  1969)  will  be 
looked  into. 
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(4)  The  implementation  and  impact  of  the  Employee  Retirement 
Income  Security  Act  of  1974  will  require  considerable  and  careful 
study  which  the  Subcommittee  expects  to  undertake. 

(5)  The  Subcommittee  also  hopes  to  give  some  attention  to  the  re- 
lationships among  the  Davis-Bacon,  "Walsh-Healey  and  Service  Con- 
tract Acts. . 

(6)  The  operation  and  assistance  by  the  United  States  of  overseas 
schools  attended  by  the  dependents  of  U.S.  officials  and  military  per- 
sonnel stationed  abroad  will  continue  to  be  the  subject  of  oversight 
activities. 

SUBCOMMITTEE  OX   MANPOWER,   COMPENSATION,  AND  HEALTH 
AND    S.YTETY 

General 

The  Subcommittee  has  jurisdiction  over  the  areas  of  occupational 
safety  and  health,  compensation  and  manpower  programs,  including 
training,  public  service  employment,  unemployment  and  full  employ- 
ment programs,  special  unemployment  assistance  and  job  placement. 

Specific 

The  Subcommittee  plans  oversight  and  investigative  activities  in 
the  following  areas: 

(1)  Manpower — the  Subcommittee  is  alreadj'  looking  into  the  ad- 
ministration and  implementation  of  the  public  service  jobs  program 
and  the  Comprehensive  Employment  and  Training  Act  with  a  view 
to  early  legislation  in  this  area.  It  will  also  conduct  investigations  into 
the  WIN  Program  and  the  job  placement  provisions  of  the  Wagner- 
Peyser  Act. 

(2)  Occupational  safety  and  health. 

(3)  Federal  Employees  Compensation  Act. 

(4)  The  relationship  of  the  Metallic  and  Nonmetallic  Mine  Safety 
Act  to  the  Coal  Mine  Health  and  Safety  Act  and  the  Occupational 
Safety  and  Health  Act  will  be  studied. 

(5)"  The  operations  of  private  employment  agencies  will  be  in- 
vestigated. 

(6)  The  Longshoremen's  and  Harbor  Workers'  Compensation  Act. 

SUBCOMMITTEE    ON    SELECT    EDUCATION 

General 

The  Subcommittee  has  diverse  jurisdiction  over  a  number  of  special 
education  programs  including  such  things  as  education  of  the  handi- 
capped, environmental  and  alcohol  and  drug  abuse  education.  It  also 
has  jurisdiction  over  a  number  of  programs  providing  services  to  the 
elderly,  the  handicapped,  the  needy,  child  development,  and  a  variety 
of  programs  related  to  art,  culture  and  libraries. 

Specific 

(1)  The  Subcommittee  will  investigate  the  applicability  of  space  age- 
research  and  technology  to  the  needs  of  handicapped  persons. 

(2)  The  Subcommittee  will  investigate  the  administrative  practices 
and  responsibilities  of  national.  State  and  local  agencies  in  the  admin- 
istration of  social  service  programs  under  its  jurisdiction. 
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(3)  The  Subcommittee  will  continue  the  study  of  applying  engineer- 
ing technology,  the  use  of  computers  and  media  devices  to  a  variety  of 
educational  programs. 

(4)  The  impact  oi'  federal  tax  laws  on  the  handicapped,  the  elderly 
and  families  in  need  of  child  care  will  be  studied. 

(5)  The  Subcommittee  will  study  the  quality  of  the  social  services 
being  provided  by  the  program  under  its  jurisdiction  and  conduct 
a  study  in  depth  of  the  recipients  of  those  services  to  develop  a  clear 
picture  of  who  they  are  and  whether  the  intended  l>eneficiaries  are 
receiving  services  as  intended. 

SUBCOMMITTKE  ON*  POSTSECON'DARY  EDUCATION* 

General 

The  Subcommittee  has  jurisdiction  over  education  beyond  the  high 
school  level. 
Specific 

(1)  The,  Subcommittee  will  continue  its  major  effort  at  oversight 
relating  to  the  student  aid  provisions  of  the  Higher  Education  Act 
which  it  expects  will  result  in  legislation  in  the  first  Session  of  this 
Congress  to  improve  the  operation  of  the  Act,  to  ease  the  default 
problem  in  student  lending  and  to  reduce  the  degree  to  which  present 
student  financial  aid  policies  encourage  tuition  increases. 

(2)  The  Subcommittee  expects  to  conduct  oversight  hearings  in 
furtherance  of  the  Committee's  functions  under  Section  431(d)  of 
the  General  Education  Provisions  Act  requiring  all  standards,  rules, 
regulations  and  requirements  of  general  applicability  issued  by  the 
Commissioner  of  Education  to  be  submitted  to  the  Congress  forty- 
five  days  prior  to  their  effective  date  during  which  the  Congress,  hav- 
ing reviewed  the  regulations,  may  return  them  to  the  agency  for 
revision. 

('.))  The  Subcommittee  intends,  in  addition,  to  examine  student 
aid  programs  in  a  number  of  areas  where  it  has  secondary  jurisdic- 
tion to  ascertain  possible  divergences  in  federal  policy  toward  assist- 
ance levels.  Among  the  programs  that  may  be  looked  into  are  the 
student  aid  programs  of  the  Public  Health  Service,  the  Law  Enforce- 
ment Assistance  Administration,  the  Department  of  Transporta- 
tion, the  Environmental  Protection  Agency,  the  Veterans'  Admin- 
istration, the  Bureau  of  Indian  Affairs,  the  Department  of  State,  the 
Maritime  Commission,  the  Social  Security  Administration,  the  Na- 
tional Science  Foundation,  the  National  Endowment  for  the  Human- 
ities and  others. 

(4)  Other  possible  areas  of  oversight  investigation  being  con- 
sidered include  consumer  protection  in  postsecondary  education  and 
the  protection  of  student  athletes. 

SUBCOMMITTEE  ON*  EQUAL  OPPORTUNITIES 

General 

The  Subcommittee  lias  general  responsibility  for  reviewing  the 
administration  and  effectiveness  of  programs  authorized  under  the 
Economic  Opportunity  Act;  the  Juvenile  Justice  and  Delinquency 
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Prevention  Act  of  1974;  title  VII  of  the  Civil  Eights  Act  of  1964; 
and  the  Domestic  Volunteer  Services  Act  of  1973  as  well  as  other 
matters  related  to  poverty  and  equal  employment  opportunities. 

Specif c 

(1)  The  Subcommittee  plans  to  oversee  implementation  of  the 
Headstart.  Economic  Opportunity  and  Community  Partnership  Act 
of  1974.  including  the  transfers  of  programs  from  the  Community 
Services  Administration;  the  effectiveness  of  the  Community  Eco- 
nomic Development  Program;  the  provision  of  services  to  handi- 
capped children  in  the  Headstart  Program;  and  a  study  of  the 
impact  of  the  reduction  of  the  federal  share  for  community  action 
programs. 

(2)  The  Subcommittee  plans  oversight  on  equal  employment  oppor- 
tunities in  four  general  areas:  (a)  oversight  of  the  Equal  Employment 
Opportunity  Commission,  particularly  its  backlog;  (b)  coordination 
between  federal  agencies  with  employment  discrimination  enforce- 
ment responsibilities:  (c)  the  effectiveness  of  the  Office  of  Federal 
Contract  Compliance  in  administering  Executive  Order  11246  barring 
employment  discrimination  by  federal  contractors;  and  (d)  employ- 
ment discrimination  in  federal  agencies. 

(#)  The  Subcommittee  plans  investigation  and  oversight  also  into 
the  ACTION  Agency's  implementation  of  domestic  volunteer  pro- 
grams :  the  failure  of  the  Law  Enforcement  Assistance  Administration 
in  the  Department  of  Justice  to  implement  the  Juvenile  Justice  and 
Delinquency  Prevention  Act  of  1974;  and  the  impact  of  the  nation's 
tax  policies  on  the  poor. 

SUBCOMMITTEE    ON    AGRICULTURAL    LABOR 

(Jencral 

The  Subcommittee  has  oversight  responsibility  for  all  laws  which 
affect  migrant  and  seasonal  farmworkers  and  their  dependents. 
Specif  c 

(1)  The  Subcommittee  plans  to  oversee  the  implementation  of  the 
recently  enacted  amendments  to  the  Farm  Labor  Contractor  Registra- 
tion Act  of  196o  by  the  Department  of  Labor. 

(2)  The  Sul>eommittee  plans  to  review  and  oversee  the  policies  and 
practices  of  the  Department  of  Labor  in  its  certification  of  the  use  of 
offshore  agricultural  workers  and  the  impact  of  such  certification  on 
the  agricultural  labor  market. 

(3)  The  Subcommittee  plans  to  oversee  the  transfer  from  the  De- 
partment of  Labor  and  the  implementation  by  the  Community  Serv- 
ices Administration  of  title  IH-b.  developmental  programs  for  mi- 
grant and  seasonal  farmworkers  (Headstart.  Economic  Opportunitv 
and  Community  Partnership  Act  of  1974). 

(4)  The  Subcommittee  will  review  the  impact  and  implementation 
of  provisions  of  existing  laws  administered  by  the  Department  of 
Health.  Education,  and  Welfare  whieh  affect  the  health  and  education 
of  migrant  and  seasonal  farmworkers. 

With  kindest  regards, 
Sincerely, 

Carl  D.  Perkixs. 
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COMMITTEE  ON  FOREIGN  AFFAIRS 


ITorsK  of  Representatives, 
CoMMirm-:  ox  Foreign*  Affairs, 
Washington^D.C:,  February  27.  Wj. 
TIon.  Jack  Brooks. 

('hainrwu.  Committee  on  Government  Operations,  House  of 
He  preventatives.  Washington,  D.C. 
Dear  Mr.  Chairman  :  Attached  is  material  coiKerniii'g  the  oversight 
activities  of  the  Foreign  Affairs  Committee  which  is  being  provided 
to  you  in  accord  with  Clause  2(c)  of  the  Rules  of  the  House  for  the 
04-th  Congress. 

I  hope  you  will  hud  this  information  responsive  to  your  request 
as  outlined  by  your  staff. 
"With  best  wishes.  I  remain 
Sincerely  yours, 

Thomas  E.  Moruax.  Chairman. 

oversight  activities  of  the  foreign  affairs  committee. 
9  4tii  coxoress 

The  ovei-sight  activities  of  the  Foreign  Affairs  Committee  in  the 
!>4th  Congress  will  extend  over  the  broad  range  of  international  affairs 
of  interest  to  or  connected  with  the  United  States.  These  activities 
will  l>e  pursuant  to  the  responsibilities  assigned  to  the  Committee 
under  the  House  rules.  Rule  X.  l(i)  lists  jurisdiction  for  the  Com- 
mittee starting  with  "Regulations  of  the  United  States  with  Foreign 
Nations  Generally"  and  includes  16  further  categories.  In  addition, 
special  oversight  functions  are  provided  in  Clause  3(d)  of  Ride  X. 

By  subject  grouping,  legislation  and  situations  under  Committee 
oversight  include:  international  relations;  conduct  of  foreign  policy; 
international  agreements;  international  and  regional  security  arrange- 
ments; interventions  abroad;  intelligence  activities  relating  to  foreign 
policy;  diplomatic  privileges  and  immunities;  operations  of  foreign 
affairs  agencies:  foreign  humanitarian,  food,  economic  and  military 
assistance:  foreign  loans:  information,  scientific,  and  cultural  ex- 
change; arms  control;  U.S.  property  abroad:  boundary  activities; 
international  organizations,  conferences  and  congresses;  protection  of 
American  citizens  and  business  interests  abroad:  international  eco- 
nomic policy:  international  financial  and  monetary  organizations: 
international  commodity  agreements  (other  than  sugar)  and  trade; 
international  fishing  agreements:  customs  administration:  and  inter- 
national education. 

The  Committee  performs  its  oversight  through  activities  of  the  full 
Committee,  a  Subcommittee  on  Oversight,  and  nine  other  functional 
(37) 
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subcommittees,  utilizing  the  Committee  staff  and  special  studies  per- 
formed for  the  Coiumittee  or  its  subcommittees  by  such  organizations 
as  the  General  Accounting  Office,  the  Congressional  Research  Service 
and  by  special  consultants. 

Specific  oversight  activities  now  underway  or  planned  by  the  full 
Committee  or  Subcommittees  include,  review  of: 

U.S.  interests,  policies,  aid,  and  commitments  relating  to  Southeast 
Asia;  POW/MIA  issues;  Current  U.S.  role.  U.S.  commitments  and 
peace  efforts  relating  to  the  Middle  East;  Demilitarization  in  the 
Middle  East;  U.S.  aims  supply  policies  and  activities  of  U.S.  defense 
contractoi-s  in  the  Persian  Gulf  area;  Palestinian  situation;  Restric- 
tions on  U.S.  aid  to  Turkey;  U.S.  policy  toward  Portugal  and  impli- 
cations for  NATO;  Spanish  base  rights  agreement;  Panama  Canal 
treaty  negotiations;  Europe,  detente,  and  U.S.  forces;  American  policy 
toward  Ethiopia;  Military  assistance,  sales  and  arms  transfer  pro- 
grams; Role  of  military7  assistance  advisory  groups;  Use  of  private 
contract  employees  for  military  training  overseas;  Intelligence  activi- 
ties relating  to  foreign  policv;  Nuclear  proliferation  and  compliance 
witli  NPT; 

U.S.  export  of  nuclear  technology;  SALT  IT.  MRFE,  (XT),  and 
oilier  arms  control  and  disarmament  negotiations;  Implications  of 
new  conventional  weapon  technology  for  national  security  and  arms 
control;  Interaction  of  science  and  technology  with  U.S.  foreign 
policy;  Executive  agreements;  Implementation  of  P.L.  93-148,  the 
War  Powers  Resolution;  Congress'  role  in  foreign  policy;  Foreign 
A  Hairs- related  required  reports  to  Congress;  Advisory  commissions 
in  I  he  foreign  policy  field;  State  Department  operations;  Foreign 
Building  operations;  U.S.  Information  Agency  operations;  Peace 
Corps  operations;  A 01) A  operations;  Foreign  service  personnel; 
Foreign  Disaster  Assistance  Programs; 

International  Aid  to  the  Sahel  Region;  U.S.  aid  to  the  drought- 
stricken  nations  of  Africa;  All)  implementation  of  Congressional  ly 
mandated  "new  directions''  in  foreign  aid  policy;  U.S. -supported 
population  programs;  Implementation  of  food  production  assistance; 
Follow-up  to  the  Workl  Food  Conference;  Foreign  policy  aspects  of 
IMi  ISO;  Overseas  food  distribution  under  PL  4H0;  Use  of  foreign 
currency  proceeds  derived  from  PL  480;  Food  policies  of  countries 
receiving  U.S.  food  aid;  Food  problems  of  developing  countries;  Ac- 
cumulation and  disposal  of  local  currencies:  Overseas  housing  guar- 
anly  program;  Operations  of  Overseas  Private  Investment  Corpora- 
tion; Operations  of  Inspector  General.  Foreign  Assistance;  Inter- 
national narcotics  traffic;  Effectiveness  of  operations  of  U.N.  systems 
and  the  I '.S. -performance  in  that  connection; 

Operations  of  certain  U.N.  specialized  agencies,  including 
UNESCO;  Human  Rights,  U.S.  foreign  policy  related  to  human 
rights,  and  the  U.X.  Human  Rights  Commission;  U.N.  Environment 
Program;  International  Conferences,  including  the  Law  of  the  Sea 
Cnnfeieiice;  Seabed  resources;  Operations  of  international  Imnking 
institution*  to  which  the  U.S.  contributes:  Ex-Tm  Rank  operations; 
Impact  of  international  monetary  issues  on  the  American  economy: 
Pi  rforniaiicc  of  the  ( 'ouncil  on  International  Economic  Policy;  Moni- 
toriii"  of  foreign  invcslmenl  in  11.2$.  bv  federal  agencies, of  American 
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investments  in  foreign  countries,  and  the  impact  on  the  U.S.  economy; 
International  codes  of  conduct  for  multinational  corporations;  Impact 
of  cartels,  monopolies  and  multinational  corporations  on  the  economy; 
International  commodity  agreements  (except  sugar);  Progress  of 
multilateral  trade  negotiations:  U.S.  government-business  relations 
affecting  the  U.S.  position  in  international  trade  and  investment; 
Political  and  economic  impact  of  the  world  oil  crisis; 

Impact  of  the  energy  crisis  on  developing  countries;  International 
Energy  Agejicy;  $25  billion  energy  safety  net;  Non-military  uses  of 
nuclear  energy  abroad,  including  related  IAEA  activities;  Adminis- 
tration of  export  controls;  Status  of  settled  and  unsettled  foreign 
claims ;  Strategic  export  limitations  and  operations  under  trading  with 
the  enemv  legislation;  U.S.  policy  toward  and  relationships  with 
South  Africa;  U.S.-Afrka  producer-consumer  relationships;  LT.S. 
lM>licy  toward  Latin  America  and  the  Caribbean;  Implementation  of 
legislation  authorizing  the  Tijuana  flood  control  project;  U.S.- 
Canadian arrangements  for  Great  Lakes  management;  Impact  of 
Canadian  actions  on  U.S.  telecommunications;  Means  for  improving 
efficiency  of  U.S. -funded  international  broadcasting;  Western  hemi- 
sphere immigration;  Customs  administration;  External  i-elations  of 
Puerto  Rico  and  the  Trust  Territory  of  the  Pacific  Islands. 

The  oversight  activities  listed  in  this  presentation  represent  what 
is  <  ui Tenth-  in  progren  or  planned  for  1075  and  1976.  Further  over- 
sight projects  will  be  developed  subsequently.  Because  of  the  uncer- 
tainties in  the  foreign  affairs  field,  it  is  impossible  to  give  a  complete 
listing  this  earh  in  the  94th  Congress  or  to  give  assurance  that  some 
plans  may  not  change  as  a  l-esult  of  future  developments  and  the  need 
to  divert  our  manpower  resources  to  more  urgent  projects. 
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Although  it  has  some  major  legislative  responsibilities,  the  primary 
function  of  the  Committee  on  Government  Operations,  as  provided  in 
the  House  rules,  is  to  "review  and  study,  on  a  continuing  basis,  the 
operation  of  Government  activities  at  all  levels  with  a  view  to  deter- 
mining their  economy  and  efficiency." 

It  carries  out  this  responsibility  through  seven  subcommittees,  each 
with  jurisdiction  over  specific  agencies  and  departments. 

The  Legislation  and  National  Security  Subcommittee  is  assigned 
responsibility  for  the  Departments  of  Defense  and  State,  the  Central 
Intelligence  Agency,  the  Arms  Control  and  Disarmament  Agency,  the 
General  Accounting  Office,  most  functions  of  the  Executive  Office  of 
the  President.  Selective  Service,  and  the  Renegotiation  Board. 

The  Intergovernmental  Relations  and  Human  Resources  Subcom- 
mittee watches  over  the  Department  of  Health.  Education,  and  Wel- 
fare, most  activities  of  the  Department  of  Agriculture,  the  Commodity 
Credit  Corporation.  Farm  Credit  Administration,  and  Federal  Crop 
Insurance  Corporation :  the  National  Science  Foundation  and  all  agen- 
cies and  programs  involving  interrelationships  between  the  Federal 
Government  and  other  levels  of  government,  of  which  the  most  im- 
portant is  revenue  sharing. 

The  Conservation.  Energy,  and  Natural  Resources  Subcommittee 
has  responsibility  for  the  Department  of  the  Interior,  the  Corps  of 
Engineers,  the  Forest  Service,  the  Environmental  Protection  Agency 
and  Council  on  Environmental  Quality,  and  all  agencies'  energy  re- 
search and  development,  including  the  Rural  Electrification  Admin- 
istration and  Tennessee  Valley  Authority. 

The  Government  Activities  and  Transportation  Subcommittee  has 
jurisdiction  over  the  Department  of  Transportation.  Civil  Aeronautics 
Board.  Federal  Maritime  Commission.  Interstate  Commerce  Commis- 
sion. National  Transportation  Safety  Board.  St.  Lawrence  Seaway 
Corporation,  and  National  Aeronautics  and  Space  Administration,  as 
well  as  the  General  Services  Administration.  Office  of  Telecommuni- 
cations Policy,  Federal  Property  Act.  and  Federal  Property  Council. 

The  Coommerce,  Consumer,  and  Monetary  Affairs  Subcommittee 
deals  with  the  Departments  of  Treasury  and  Commerce,  the  Federal 
Trade  Commission.  Small  Business  Administration.  Federal  Reserve 
System.  Council  of  Economic  Advisers,  and  the  many  agencies  in- 
volved in  commerce,  consumer,  and  monetary  affairs. 

The  Manpower  and  Housing  Subcommittee  has  the  Departments  of 
Labor  and  Housing  and  Urban  Development  as  its  prime  responsi- 
bility, but  also  oversees  the  Veterans'  Administration.  Civil  Service 
Commission,  the  Office  of  Economic  Opportunity,  and  several  com- 
missions and  boards  that  operate  in  the  field  of  manpower  and  labor 
relations. 

The  Government  Information  and  Individual  Rights  Subcommittee 
checks  on  the  Justice  Department  and  the  Federal  courts,  the  Federal 
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Communications  Commission,  U.S.  Postal  Service,  and  statutes  deal- 
ing with  freedom  of  information  and  the  right  of  privacy,  and  the 
ngencies  that  administer  them. 

Each  of  these  subcommittees  has  submitted  its  oversight  plans  for  the 
04th  Congress  in  accordance  with  Rule  X,  Clause  2(c),  and  I  hereby 
submit  them  for  inclusion  in  this  report. 

Jack  Brooks, 

Chairman. 


LEGISLATION    AND    NATIONAL    SECURITY   8UBCOMJ 

Jurisdiction 

The  subcommittee  will  maintain  oversight  of  the  functions  and 
operations  of  the  Department  of  Defense,  State  Department,  Agency 
for  International  Development,  Executive  Office  of  the  President, 
General  Accounting  Office,  Selective  Service  System,  Central  Intelli- 
gence Agency,  Renegotiation  Board,  and  Arms  Control  and  Dis- 
armament Agency. 
Department  of  Defense 

Transportation  Data  Systems  and  Other  GAO  Reports;  Foreign 
Troop  Training  Contracts  with  Private  Corporations;  Illegal  Con- 
tracts for  Computer  Systems;  Recommendations  of  Commission  on 
Government  Procurement ;  Proper  and  Efficient  Use  of  Defense  Man- 
power; Further  Study  of  "Parts  Standardization  in  Weapon  Sys- 
tems"; Contracting-Out  Policies;  Cost  Overruns  in  Defense  Procure- 
ment. 

State- AID 

Organizational  Deficiencies  and  Staff  Morale;  AID  Policies  in  For- 
eign Assistance;  Consolidation  of  Administrative  Support  Functions; 
Collection  of  Delinquent  Debts. 

Intelligence 

Overlapping  and  Duplication  Among  Agencies. 
Other  Areas 

Office  of  Federal  Procurement  Policy;  Office  of  Management  and 
Budget — Domestic  Council. 

Jack  Brooks,  Chairman. 

intergovernmental  relations  and  human  resources  subcommittee 

House  of  Representatives, 
Washington,  D.C.,  March  7, 1976. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government.  Operations,  2157  Raybvm 
House  Office  Building.  Washington,  D.C. 
Dear  Jack  :  In  response  to  your  request  of  February  24,  I  am  en- 
closing a  brief  listing  of  the  more  significant  oversight  activities  pres- 
ently contemplated  tor  the  94th  Congress.  Almost  all  of  these  matters 
are  currently  under  active  investigation. 

Other  matters  will  no  doubt  be  added  to  our  oversight  agenda 
later  on  the  basis  of  events  taking  place  during  the  next  two  years. 
Sincerely, 

L.  H.  Fountain,  Chairman. 
Enclosure. 
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.  Qvevsight  activities  presently  contemplated  during  the  94th  Con- 
gress incmdc  Investigation  and/or  study  of: 

Department  of  Health,  Education,  and  Welfare 

Procedures  and  resources  utilized  by  the  Department  of  Health, 
tiducition,  and  Welfare  in  the  investigation  of  fraud  and  other  pro- 
gram irregularities. 

Administration  of  Social  Security  programs,  with  particular  em- 
phasis on  problems  involving  the  Supplemental  Security  Income 
(.SSI)  program. 

Administration  of  the  Aid  to  Families  with  Dependent  Children 
(AFDC)  program. 

Administration  of  the  Medicare  and  Medicaid  programs. 

Administration  of  the  Guaranteed  Student  loan  program  and  other 
student  financial  aid  programs  administered  by  the  Office  of 
Education. 

.  Regulatory  policies,  procedures  and  activities  of  the  Food  and  Drug 
Administration  for  assuring  the  safety  and  effectiveness  of  new  drugs. 

Effectiveness  of  Federal  programs  intended  to  combat  cancer  and 
heart  diseases. 

Possible  irregularities  in  travel  expenditures  by  officials  of  the 
Department  of  Health,  Education,  and  Welfare. 

Department  of  Agriculture 

Rural  housing  operations  of  the  Farmers  Home  Administration. 

Selected  aspects  of  the  regulation  of  commodity  futures  trading  by 
the  Commodity  Exchange  Authority  and  successor  agencies. 

Selected  aspects  of  the  administration  of  the  Food  Stamp  program. 

Activities  of  the  Department  of  Agriculture  relating  to  exports  of 
agricultural  commodities. 

Intergovernmental  Relations 

The  General  Revenue  Sharing  program  established  by  the  State 
and  Local  Fiscal  Assistance  Act  of  1972  (P.L.  92-512). 

conservation,  energy,  and  natural  resources  subcommittee 

House  of  Representatives, 
Washington,  D.C.,  March  3,  1,975. 
Hon.  Jack  Brooks, 

Chairman,    Committee    on,    Government    Operations,    Room   2157, 
liayburn  Building,  Washington,  D.C. 

Dear  Mr.  Chairman:  This  is  in  response  to  your  request  of  Feb- 
ruary 24  for  a  relatively  concise  listing  of  oversight  activities  con- 
templated by  the  Conservation,  Energy,  and  Natural  Resources  Sub- 
committee during  the  present  Congress. 

All  of  the  items  listed,  of  course,  are  subject  to  approval  and  pri- 
ority determination  by  the  Subcommittee  when  it  holds  its  organiza- 
tional meeting  this  month.  Your  suggestions  also  are  actively  solicited. 

With  kind  regards. 
Sincerely, 

William  S.  Moorhead,  Chairman. 
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Jurisdiction 

The  Subcommittee  maintains  oversight  of  the  activities  and  opera- 
tions of  the  following  agencies  and  departments :  Environmental  Pro- 
tection Agency,  Council  on  Environmental  Quality,  Energy  Research 
and  Development  Administration,  Federal  Energy  Administration, 
Federal  Power  Commission,  Nuclear  Regulatory  Commission,  Rural 
Electrification  Administration,  Tennessee  Valley  Authority,  Depart- 
ment of  the  Interior,  Corps  of  Engineers,  Forest  Service,  and  Indian 
Claims  Commission. 
Oversight 

Investigative  program  and  priority  order  subject  to  approval  by 
subcommittee. 

Federal  involvement  of  the  above  agencies  in  particular  and  of  the 
Executive  branch  in  general  in  the  following  activities : 

1.  Development  of  oil  shale  deposits  and  on-shore  oil  and  gas  leases ; 

2.  Conversion  of  power  generators  to  coal  and  other  fuels; 

3.  New  energy  sources  offering  potential  major  benefits  in  the  near 
future ; 

4.  Construction  of  new  gasification  plants  utilizing  coal,  oil  shale 
and  garbage; 

5.  Solid  wastes,  including  conservation,  environmental  and  energy 
aspects ; 

6.  Assisting  State  and  local  governments  to  obtain  needed  informa- 
tion on  new,  high-benefit  energy,  conservation  and  environmental 
advances  which  they  could  adopt ; 

7.  Duplication  and  overlapping  of  energy,  conservation  and  en- 
vironmental operations; 

8.  National  Park  concession  practices  (continue  review  initiated 
last  session)  ; 

9.  Pass-through  of  higher  energy  costs  to  consumers  by  public 
utilities ; 

10.  Power  Rate  Increases  (Bureau  of  Reclamation,  Central  Valley 
Project,  California) ; 

11.  Enforcement  of  Federal  Mine  Health  and  Safety  Laws; 

12.  Garrison  Diversion  Unit,  North  Dakota ; 

13.  Study  of  automotive  fuel  economy ; 

14.  Administration  of  the  Fish  and  Wildlife  Coordination  Act; 
and 

15.  Other  items  yet  to  be  determined  by  future  events,  GAO  reports, 
Member  requests  and  new  developmente  in  past  investigations. 

government  acitvittes  and  transportation  subcommittee 

House  of  Representatives, 
Washington,  D.C.,  March  10, 1975. 
Hon.  Jack  Brooks, 

Chairman,   Committee   on   Government   Operations,  2157  Raybum 
House  Office  Building,  Washington,  D.C. 
Dear  Mr.  Chairman  :  This  is  in  response  to  your  request  of  Febru- 
ary 24,  1975,  for  a  listing  of  oversight  activities  contemplated  by  the 
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•Government  Activities  and  Transportation  Subcommittee  during  the 
present  Congress. 

The  listed  items  are,  of  course,  subject  to  approval  and  priority 
determination  by  the  Subcommittee  when  it  holds  its  organizational 
meeting. 

Sincerely, 

Wm.  J.  Randall, 
Chairman,  Government  Activities  and 

Transportation  Subcommittee. 
Enclosure. 

.Assigned  jurisdiction 

Transportation  Matters. — Department  of  Transportation  and  com- 
ponent agencies  including  the  U.S.  Coast  Guard  and  Saint  Lawrence 
•Seaway  Development  Corporation;  Civil  Aeronautics  Board;  Federal 
Maritime  Commission;  Interstate  Commerce  Commission;  National 
Aeronautics  and  Space  Administration;  National  Transportation 
Safety  Board. 

Government  Activities. — General  Services  Administration ;  Federal 
Property  Act;  Federal  Property  Council;  Office  of  Telecommunica- 
tions Policy. 
Proposed  oversight  activities 

1.  Air  transportation  of  hazardous  cargo. 

2.  Conflicting  altitude-control  responsibility  during  aircraft  land- 
ing approach. 

3.  Requirements  for  altitude  monitoring  avionic  equipment  in  com- 
mercial aircraft. 

4.  Review  of  certain  procurement  actions  by  the  Department  of 
Transportation  and  Federal  Aviation  Administration  involving  land- 
ing-aid equipment. 

5.  Coast  Guard  search  and  rescue  operations. 

6.  Utilization,  operation,  and  management  of  the  St.  Lawrence  Sea- 
way. 

7.  Interstate  Commerce  Commission  responsiveness  to  rate-change 
requirements  in  the  public  sector. 

8.  Utilization  and  disposal  of  Federal  excess  and  surplus  property 
(real  and  personal),  including  negotiated  disposals.  Consideration  of 
legislation  to  improve  and  broaden  the  Federal  programs  for  donating 
surplus  property  and  transferring  excess  property. 

9.  Extent  of  compliance  with  computer  procurement  and  manage- 
ment provisions  of  section  111  of  the  Federal  Property  Act,  which 
provides  for  the  economic  and  efficient  purchase,  lease,  maintenance, 
operation,  and  utilization  of  automatic  data  processing  equipment  by 
Federal  agencies. 

commerce,  consumer,  and  monetary  affairs  subcommittee 

House  of  Representatives, 
Washington,  D.C.,  March  11, 1975. 
Hon.  Jack  Brooks, 

-Chairman,  Government  Operations  Committee,  Rayburn  House  Office 
Building,  Washington,  D.C. 
Dear  Mr.  Chairman  :  This  is  in  response  to  your  letter  of  Febru- 
ary 27,  1975,  requesting  a  report  from  the  Commerce,  Consumer,  and 
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Monetary  Affairs  Subcommittee  on  its  oversight  agenda  for  the  94th 
Congress. 

The  oversight  agenda  outlined  below  establishes  areas  of  potential 
Subcommittee  investigation  that  I  regard  as  most  critical.  It  is,  of 
course,  axiomatic  that  the  interests  of  subcommittee  members  or  future 
events  might  dictate  a  departure  from  this  agenda. 

I.    BANKING 

(A)  Responsibility  for  the  regulation  and  supervision  of  federally 
chartered  or  federally  insured  banks,  savings  and  loan  associations  and 
other  financial  institutions  has  been  entrusted  to  four  major  agencies: 
the  Federal  Reserve  Board,  the  Comptroller  of  the  Currency,  the 
Federal  Home  Loan  Bank  Board  and  the  Federal  Deposit  Insurance 
Corporation.  The  alarming  increase  in  the  numbers  of  bank  failures, 
the  increase  in  the  numbers  of  defaulted  and  foreclosed  loans  and  alle- 
gations of  unfair  and  deceptive  banking  and  other  financial  institution 
practices  suggest  the  need  tor  a  thorough  investigation  of  the  efficiency 
-and  economy  of  the  Federal  banking  regulatory  apparatus.  Certain 
critics  have  suggested,  for  example,  that  regulatory  responsibility  is 
excessively  fragmented  and  that  bank  examination  procedures  are 
inadequate; 

(B)  Of  all  the  Federal  regulatory  agencies,  the  financial  regulatory 
•entities  seem  to  operate  in  a  way  that  fosters  a  minimum  amount  of 
public  accountability  and  a  maximum  amount  of  secrecy.  An  over- 
sight investigation  into  the  efficiency  and  accessibility  of  Federal  bank- 
ing agency  procedures  and  practices  would  seem  to  be  indicated.  Of 
particular  concern  to  the  Subcommittee  is  whether  the  personnel  rela- 
tionships between  the  Federal  regulatory  entities  and  the  institutions 
they  regulate  raise  conflict-of-interest  questions;  and  consumer  infor- 
mation practices  and  procedures  of  the  Federal  banking  agencies  are 
adequate. 

II.    CONSUMER    PROTECTION    AND    INDUSTRIAL    CONCENTRATION 

The  Federal  consumer  protection  apparatus,  particularly  the  Fed- 
eral Trade  Commission's  authority  to  deal  with  unfair  and  deceptive 
acts  and  practices  in  commerce,  determines,  to  a  large  extent,  whether 
the  free  enterprise  system  will  operate  in  a  manner  that  is  not  only 
efficient  but  just.  Similarly,  the  success  of  the  free  enterprise  system 
depends  in  large  measure  on  the  degree  to  which  Federal  programs, 
such  as  those  of  the  Commission's  Bureau  of  Competition,  foster 
maximum  competition  in  key  industrial  areas.  Recently,  serious  ques- 
tions have  been  raised  about  the  efficiency  of  FTC  rules  and  procedures 
leading  to  anti-trust  actions  and  the  promulgation  of  trade  regulation 
rules.  If  Commission  rules  and  procedures  impede  consumer  protection 
and  the  resolution  of  possible  anti-competitive  practices,  then  it  is 
incumbent  on  the  Subcommittee  to  examine  apparent  deficiencies. 
Moreover,  questions  have  also  been  raised  regarding  the  efficiency  and 
economy  of  FTC  operations  relative  to  assuring  the  truthfulness  and 
integrity  of  energy  and  other  advertising. 

III.  FEDERAL  GOVERNMENT^  ECONOMIC  DECISIONMAKING  PROCESS 

Fiscal,  monetary  and  regulatory  decisions  and  policies  by  the  Federal 
Government  determine,  to  a  significant  extent,  the  success  or  failure  of 
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our  economic  system.  Nevertheless,  there  is  enormous  proliferation  andl 
fragmentation  within  the  economic  decision-making  processes  of  gov- 
ernment. Serious  questions  have  been  raised  regarding  the  ability  of 
our  far-flung  governmental  economic  apparatus  to  meet  and  be  re- 
sponsive to  the  challenges  of  our  times.  While  numerous  congressional, 
executive  branch  and  other  outside  investigations  have  focused  on  the 
prudence  and  efficacy  of  various  substantive  economic  policies,  no  over- 
sight has  been  conducted  into  the  organizational  apparatus  that  ulti- 
mately formulates  economic  policy  for  the  United  States.  Such  an 
investigation  is  clearly  indicated. 

IV.   INTERNATIONAL  TRADE 

Our  Nation's  current  difficulties  with  inflation,  recession  and  short- 
ages in  key  commodity  areas  and  the  increased  political  and  economic 
importance  of  international  trade,  all  point  up  the  need  for  an  over- 
sight review  of  the  efficiency  and  economy  of  our  international  trade 
policies,  practices  and  procedures.  The  failure  of  our  Federal  interna- 
tional trade  organizational  apparatus  to  anticipate  the  effect  on  Ameri- 
can consumers  of  unbridled  shipments  abroad  of  wheat  and  key  feed 
grains ;  issues  involving  trade  in  strategic  materials ;  and  allegations 
of  U.S.  governmental  complicity  in  boycotts  and  blacklisting  based  on 
religious  heritage,  all  point  up  the  need  for  appropriate  congressional 
oversight  in  this  area. 

V.    SECURITIES   AND  EXCHANGE   COMMISSION/CORPORATE  RESPONSIBILITY 

Much  attention  has  been  devoted  by  committees  of  the  Congress  and 
industry  trade  groups  to  the  fair  and  efficient  functioning  of  our  se- 
curities and  financial  market  system.  As  a  result  of  the  efforts  by  the 
SBC,  the  stock  exchanges  and  other  interested  groups,  the  investing 
public  has  been  provided  with  much  information  and  malpractices 
have  been  minimized.  But  the  Securities  and  Exchange  Commission 
has  interpreted  its  legal  mandate  in  a  way  that  tends  to  exclude  the 
issue  of  corporate  citizenship  and  corporate  responsibility.  In  an  age 
when  many  corporations  exercise  great  influences  over  the  daily  lives 
of  our  citizens  and  frequently  operate  as  miniature-governments,  it  is 
important  to  examine  the  SEC's  practices  and  procedures  for  assuring 
that  corporate  decision-making  is  made  in  the  broad  public  interest 
and  with  appropriate  public  in-put. 

VI.    INTERNAL  REVENUE   SERVICE   TAX   MATTER8 

Several  issues  involving  IRS  operations  seem  to  require  the  over- 
sight attention  of  the  Subcommittee :  The  first  has  to  do  with  the  uni- 
formity of  IRS  procedures  and  standards  for  the  informal  settle- 
ments of  tax  claims  by  and  against  the  Government.  The  second 
concerns  the  efficiency  or  IRS  procedures  designed  to  force  compliance 
with  its  prohibition  against  the  deductibility  of  business  expenses; 
associated  with  influencing  congressional  or  other  public  policies.  The- 
third  involves  the  efficiency  of  IRS  services  and  assistance  to  tax 
payers  for  the  reporting  and  filing  of  income  taxes. 
Sincerely, 

Benjamin  S.  Rosenthal,  Chairman. 
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MANPOWER  AND  HOUSING  SUBCOMMITTEE 

House  of  Representatives, 
Washington,  B.C.,  March,  6,  1975. 
Hon.  Jack  Brooks,  .         M  ._ 

Chairman,  Committee  on  Government  Operations,  2157  Rayburn 
House  Office  Building,  Washington,  D.C. 
Dear  Mr.  Chairman  :  In  accordance  with  your  request  of  Febru- 
ary 24th,  I  am  enclosing  a  list  of  proposed  investigations  by  the  Man- 
power and  Housing  Subcommittee.  Two  investigations  are  already 
under  way  at  the  staff  level. 

The  priority  that  will  be  assigned  to  these  subject  areas  will  be 
determined  at'a  Subcommittee  meeting  to  take  place  later  this  month. 
I  would  add  the  usual  caveat  that  modifications  to  any  plan  will 
probably  be  instituted  as  a  result  of  future  developments  and  infor- 
mation that  comes  to  our  attention. 
Sincerely  yours, 

Floyd  V.  Hicks,  Chairman. 

Jurisdiction 

Department 
Department  of  Housing  and  Urban  Development. 
Department  of  Labor. 

Other  agencies 
Veterans'  Administration 
•Civil  Service  Commission 

Special  Action  Office  for  Drug  Abuse  Prevention 
ACTION 

Federal  Mediation  and  Conciliation  Service 
National  Labor  Relations  Board 
National  Mediation  Board 
Office  of  Economic  Opportunity 
Railroad  Retirement  Board 
Equal  Employment  Opportunity  Commission 
Occupational  Safety  and  Health  Review  Commission 

Proposed  areas  of  oversight 

1.  Housing:  Mortgage  Industry's  adherence  to  HUD  guidelines 
designed  to  preclude  unnecessary  foreclosure  on  mortgages  held  by 
low-income  home  owners.  (Already  under  way.) 

2.  The  effectiveness  of  HUD  policies  and  programs  for  the  acquisi- 
tion, retention  and  disposition  of  properties  owned  by  the  department. 

3.  The  interrelationship  between  serious  disability  claims  and  safety 
standards  for  Federal  workers  in  areas  of  high  risk  employment.  (Al- 
ready under  way.) 

4.  Effectiveness  of  Federally  funded  State  unemployment  offices 
and  employment  services. 

5.  Labor  Department  use  and  audit  of  past  manpower  training 
contracts. 

6.  The  effectiveness  of  Government  agencies'  determination  of  per- 
sonnel requirements  and  the  use  of  non-career  employees  for  tem- 
porary requirements. 
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7.  The  effectiveness  of  Federal  programs  to  identify  and  rehabilitate 
alcoholic  workers.  (Follow-up  investigation.) 

8.  Methods  used  to  disseminate  effective  programs  to  assist  drug 
abusers. 

9.  Implementation  of  recommendations  on  the  Committee's  report 
on  "Impact  of  Heroin  Addiction  on  the  Criminal  Justice  System." 

Additional  areas  are  under  consideration  as  a  result  of  Member  in- 
quiries posed  at  the  first  Subcommittee  meeting.  Based  on  past  Con- 
gresses, it  is  likely  that  several  areas  for  investigation  will  develop 
over  the  next  few  months. 

government  information  and  individual  rights  subcommittee 

House  of  Bepresentatives, 
Washington,  D.C.,  March  5, 1975. 
Hon.  Jack  Brooks. 

Chairman,  Committee  on  Government  Operations,  Room  2157  Ray- 
burn  House  Office  Building,  Washington,  D.C. 
Dear  Mr.  Chairman  :  The  attached  is  in  reply  to  your  letter  of 
February  24,  1975,  requesting  a  concise  listing  of  contemplated  over- 
sight activities  during  the  94th  Congress  by  this  Subcommittee  pur- 
suant to  the  provisions  of  House  Rule  X,  clause  (2)  (c). 

A  work  agenda  was  discussed  and  approved  at  our  Subcommittee's 
organizational  meeting  on  February  19,  1975,  which  included  both 
oversight  and  legislative  hearings  for  the  next  several  months.  Other 
oversight  work  for  the  remainder  of  the  94th  Congress  is  projected, 
on  the  basis  of  informal  discussions  among  our  Members. 
With  best  wishes, 
Sincerely, 

Bella  S.  Abzug,  Chairwoman. 

Jurisdiction 

The  Subcommittee  maintains  oversight  and  legislative  jurisdiction 
over  the  activities  and  operations  of  all  Federal  agencies  with  respect 
to  the  Freedom  of  Information  Act  (5  U.S.C.  552),  the  Privacy  Act 
of  1974  (5  U.S.C.  552a),  and  the  Federal  Advisory  Committee  Act 
(5  U.S.C.  App.  1)  86  Stat.  770. 

It  also  maintains  oversight  jurisdiction  over  the  activities  and  opera- 
tions of  the  U.S.  Information  Agency,  the  National  Archives  and 
Kecords  Service,  the  Government  Printing  Office,  the  Federal  Com- 
munications Commission,  the  U.S.  Postal  Service,  the  Department  of 
Justice,  the  U.S.  Courts,  the  Civil  Rights  Commission,  the  Smith- 
sonian Institution,  the  National  Foundation  on  the  Arts  and  the  Hu- 
manities, and  the  American  Revolution  Bicentennial  Administration. 
Oversight  targets 

1.  Monitoring  and  oversight  activities  involving  Federal  agencies' 
implementation  of  the  1974  Freedom  of  Information  Act  amendments 
(P.  L.  93-502). 

2.  Monitoring  and  oversight  activities  involving  Federal  agencies' 
implementation  of  the  Privacy  Act  of  1974  (P.  L.  93-579). 

3.  Completion  of  work  leading  to  action  by  Subcommittee  and  fulfc 
Committee  on  draft  reports  on  "Telephone  Monitoring  Practices  of 
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Federal  Agencies"  and  "Federal  Agencies'  Use  of  Polygraphs  and 
Similar  Devices."  (Based  on  1974  investigations  and  hearings  by  the 
Subcommittee.) 

4.  Preliminary  1-day  oversight  hearings  with  new  agencies  added  to 
the  Subcommittee's  jurisdiction  to  acquaint  Members  with  the  broad 
aspects  of  each  agency's  programs,  operations,  personnel,  budgetary 
requirements,  and  similar  general  information.  These  hearings  will 
include  testimony  from : 

U.S.  Information  Agency 

Department  of  Justice 

U.S.  Postal  Service 

Federal  Communications  Commission 

National  Archives  and  Records  Service 

U.S.  Civil  Rights  Commission 

National  Foundation  on  the  Arts  and  Humanities 

Smithsonian  Institution 

American  Revolution  Bicentennial  Administration 

U.S.  Government  Printing  Office 

5.  Review  of  Federal  agencies'  collection  and  dissemination  of  sci- 
entific and  technical  information,  overlapping  and  duplication  of  such 
efforts  and  their  economy  and  efficiency'. 

6.  Review  of  the  operations  of  public  information  offices  in  key  se- 
lected Federal  agencies,  including  extent  of  "contracting  out"  of  agency 
"public  relations"  activities. 

7.  Review  of  the  scope  of  planned  activities  and  programs  of  the 
American  Revolution  Bicentennial  Administration,  their  economy  and 
efficiency  and  impact  on  the  Bicentennial  Year  festivities. 

8.  Review  of  the  activities  of  the  National  Archives  and  Records 
Service,  with  particular  reference  to  their  work  involving  declassifica- 
tion of  agency  documents,  Presidential  libraries  administration,  and 
the  general  economy  and  efficiency  of  NARS  operations. 

9.  Review  of  the  economy  and  efficiency  of  operations  of  the  Federal 
Communications  Commission  and  their  various  statutory  responsibili- 
ties in  rate-making,  broadcasting,  and  other  fields. 

10.  Review  of  the  economy  and  efficiency  of  operations  of  the  U.S. 
Information  Agency,  with  reference  to  their  status  as  an  independent 
arm  of  the  State  Department  and  recommendations  in  the  forthcoming 
report  of  the  Stanton  Commission. 

11.  Various  other  in-depth  oversight  hearings  on  those  agencies 
listed  in  paragraph  4  above,  depending  on  information  developed  dur- 
ing the  preliminary  orientation  hearings,  GAO  reports,  and  other 
future  developments  that  may  make  some  of  them  high  priority  areas 
for  oversight  activity. 
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House  op  Representatives, 
committee  on  house  administration, 

Washington,  D.C.,  March  4-, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  U.S.  House  of  Rep- 
resentatives, Washington,  D.C. 

Dear  Mr.  Chairman  :  Thank  you  for  your  letter  of  February  13, 
1975  concerning  a  report  from  the  Committee  on  House  Administra- 
tion on  oversight  activities  of  this  Committee,  pursuant  to  Clause  2(c) 
of  Rule  X  of  the  Rules  of  the  House  of  Representatives. 

Because  this  overview  on  oversight  responsibilities  is  required  at  the 
beginning  of  the  Congress,  it  should  be  understood  that  the  Committee 
may  weU  expand  or  contract  its  oversight  activities  as  the  need  arises. 
It-  is  now  contemplated  Committee  oversight  activities  will  be  orga- 
nized along  subcommittee  jurisdictions  rather  than  appointing  a  spe- 
cial oversight  committee.  The  Committee  on  House  Administration 
has  nine  Standing  Subcommittees  and  two  Ad  Hoc  Subcommittees 
and  each  will  be  responsible  for  oversight  in  its  areas  of  jurisdiction. 

Oversight  activities  contemplated  by  Subcommittees  are  as  follows : 

The  Subcommittee  on  Accounts  carries  out  the  oversight  responsi- 
bility specified  in  2  USC  91  and  95,  to  ensure  the  performance  of  du- 
ties by,  and  to  review  the  remuneration  of,  officers  and  employees  of 
the  House,  and  to  oversee  payments  from  the  contingent  fund. 

The  Subcommittee  on  Elections  carries  out  the  Committee's  over- 
sight responsibilities  on  matters  relating  to  the  election  of  the  Presi- 
dent, Vice  President,  and  Members  of  Congress;  corrupt  practices, 
contested  elections;  credentials  and  qualification;  and  Federal  elec- 
tions generally. 

The  Subcommittee  on  Library  and  Memorials  has  oversight  respon- 
sibilities in  matters  relating  to  the  Library  of  Congress  and  the  House 
Library,  statuary  and  picture^  acceptance  and  purchase  of  works  of 
art  for  the  Capitol ;  the  Botanic  Gardens ;  management  of  the  Library 
of  Congress;  purchase  of  books  and  manuscripts;  erection  of  monu- 
ments to  the  memory  of  individuals  and  matters  relating  to  the  Smith- 
sonian Institution. 

The  Subcommittee  on  Printing  has  oversight  responsibilities  deal- 
ing with  the  printing  of  House  documents ;  the  Congressional  Record ; 
bills  concerning  the  Government  Printing  Office,  the  Depository  Li- 
brary Program  and  the  use  of  recycled  paper  in  government  docu- 
ments. In  addition,  the  subcommittee  will  have  oversight  responsi- 
bilities over  the  implementation  of  the  Presidential  Recordings  and 
Materials  Preservation  Act,  monitoring  the  work  of  the  National 
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Study  Commission  on  Records  and  Documents  of  Federal  Officials  and 
considering  legislation  regarding  the  disposition  of  documents  of  Fed- 
eral officials. 

The  Subcommittee  on  Electrical  and  Mechanical  Office  Equipment 
has  responsibility  for  evaluating  and  allocating  all  office  equipment 
used  by  the  House  of  Representatives  as  well  as  procurement 
programs. 

The  Subcommittee  on  Personnel  and  Police  has  oversight  responsi- 
bilities of  the  classification  and  evaluation  program  for  the  House  of 
Representatives,  the  Sergeant  at  Arms,  the  Doorkeeper,  and  the  Post- 
master. The  Subcommittee  has  further  oversight  responsibilities  with 
regard  to  the  U.S.  Capitol  Police  Board  in  administering  the  1,040 
member  force  of  the  U.S.  Capitol  Police.  It  will  also  review  the  secu- 
rity system  for  the  Capitol  complex  authorized  by  House  Concurrent 
Resolution  550,  92nd  Congress. 

The  Subcommittee  on  Contracts  has  oversight  activities  over  the 
provisions  of  section  303  of  the  Legislative  Reorganization  Act  of 
1970,  which  pertains  to  the  authorization  of  standing  committees  of 
the  House  to  obtain  the  temporary  services  of  individual  consultants 
or  organizations  to  make  studies  or  advise  the  committee  with  respect 
to  any  matter  within  its  jurisdiction. 

The  Subcommittee  on  Parking  is  charged  with  oversight  responsi- 
bilities regarding  vehicle  parking  facilities  and  the  regulating  of  park- 
ing spaces  for  Members  and  House  employees. 

The  Subcommittee  on  Paper  Conservation  has  oversight  responsi- 
bility with  regard  to  the  distribution  and  the  accumulation  of  printed 
material  in  the  House  of  Representatives. 

The  Ad  Hoc  Subcommittee  on  Computers  is  charged  with  oversight 
responsibilities  with  regard  to  the  House  Information  Systems  and  use 
of  automated  data  processing  equipment. 

The  Ad  Hoc  Subcommittee  on  the  Restaurants  has  oversight  re- 
sponsibilities with  regard  to  the  overall  food  service  facilities  of  the 
House. 

Doug  Frost,  Staff  Director,  and  Paula  Peak,  Deputy  Staff  Director, 
have  been  designated  as  representatives  of  this  Committee  to  confer 
with  Government  Operations  Committee  staff. 

If  I  can  be  of  any  further  assistance,  please  don't  hesitate  to  call 
on  me. 

With  kind  personal  regards,  I  am 
Very  sincerely  yours, 

Wayne  L.  Hays,  Chairman. 
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U.S.  House  or  Representatives, 
Committee  on  Interior  and  Insular  Affairs, 

Washington,  B.C.,  February  28, 1975. 
Hon.  Jack  Brooks, 

Chairman,   Committee  on  Government   Operations^  2157  Raybum 
House  Office  Building,  Washington,  B.C. 

Dear  Mr.  Chairman  :  This  is  in  response  to  your  request  to  provide 
information  on  oversight  activities  contemplated  by  this  Committee 
during  the  94th  Congress. 

In  carrying  out  both  the  general  oversight  responsibilities  under 
Rule  X,  Clause  2(b),  and  the  special  oversight  functions  under  Rule  X, 
Clause  2(e),  of  the  Rules  of  the  House,  oversight  as  well  as  legislative 
jurisdiction  is  assigned  under  our  Committee  Rules  to  seven  sub- 
committees in  their  respective  areas.  However,  the  Full  Committee 
may  from  time  to  time  initiate  oversight  inquiries  when  they  appear 
to  involve  the  jurisdiction  of  several  subcommittees.  In  addition,  the 
Committee  will  designate  staff  representatives  to  assist  the  subcom- 
mittees in  fulfilling  their  oversight  role  and  to  work  with  the  Com- 
mittee on  Government  Operations. 

Rule  X,  Clause  (1),  of  the  Rules  of  the  House  sets  forth  the  juris- 
diction of  this  Committee.  Rule  16  of  the  Rules  of  the  Committee  on 
Interior  and  Insular  Affairs  further  assigns  these  area6  of  general 
Committee  jurisdiction  to  the  respective  subcommittees.  A  copy  of  the 
Committee  Rules  is  enclosed,*  and  you  will  note  on  page  9  that  the 
special  oversight  function  with  respect  to  non-military  nuclear  energy 
and  research  and  development,  including  the  disposal  of  nuclear  waste, 
is  given  to  the  Subcommittee  on  Energy  and  the  Environment;  and 
that  the  special  oversight  function  with  respect  to  all  programs  affect- 
ing Indians  goes  to  the  Subcommittee  on  Indian  Affairs. 

Exercise  of  the  oversight  responsibilities  and  functions  of  this  Com- 
mittee were  initiated  this  month  by  oversight  inquiries  into  the  Alaska 
pipeline  construction  by  the  Subcommittee  on  Public  Lands  and  these 
are  expected  to  continue.  On  March  4,  oversight  hearings  will  be  held 
by  the  Subcommittee  on  Mines  and  Mining  on  Bureau  of  Mines  and 
U.S.  Geological  Survey  programs.  On  March  5  an  orientation  meeting 
with  the  Secretary  of  the  Interior  has  been  scheduled  by  the  Full 
Committee.  The  Full  Committee  sessions  will  be  followed  by  hearings 
March  6,  7,  13,  and  14  on  national  outdoor  recreation  programs  and 
policies  to  be  held  by  the  Subcommittee  on  National  Parks  and  Recrea- 

•Document  available  In  flies  of  the  Committee  on  Government  Operations. 
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tion.  Later,  on  March  14,  an  orientation  meeting  with  the  Bureau  of 
Reclamation  personnel  has  been  scheduled  by  the  Subcommittee  on 
"Water  and  Power  Resources. 

Although  our  detailed  schedule  is  not  yet  complete  for  the  remainder 
•of  this  Congress,  we  do  contemplate  in  addition  to  those  areas  to  which 
I  referred  oversight  activities  by  the  Subcommittees  as  follows : 

National  Parks  and  Recreation. — The  administration  of  the  National 
Trails  Systems  Act,  the  Pennsylvania  Avenue  Development  Plan  for 
the  nation's  Bicentennial  celebration,  and  concessions  in  the  National 
Park  System. 

Water  and  Power  Resources. — The  implementation  of  P.L.  84-984 
and  P.L.  84-100,  having  to  do  with  small  project  loans  and  distribu- 
tion system  loans;  the  Interior  Department's  research  program;  and 
the  status  of  feasibility  studies  that  Congress  has  authorized  in  recent 
years. 

Energy  and  the  Environment. — Nonmilitary  nuclear  energy,  re- 
search and  development,  and  the  disposal  of  nuclear  waste. 

Territorial  and  Insular  Affairs. — Marianas  status  negotiations  and 
examination  of  non-essential  elements  of  the  organic  acts  for  Guam 
and  the  Virgin  Islands. 

Mines  and  Mining. — Strip  mining  regulations  on  Federal  lands  as 
well  as  other  oversight  functions  being  carried  out  as  adjuncts  to  hear- 
ings on  major  legislation  before  the  Subcommittee. 

Indian  Affairs. — The  administration  of  the  leasing  of  Indian  trust 
lands,  and  tribal  governmental  problems  in  general. 

Public  Lands. — Oil  and  gas  production  from  public  lands  (deter- 
mine extent  of  "shut-in'"  capacity) ;  Bureau  of  Land  Management 
( BLM)  and  Forest  Service  mining  regulations  as  they  apply  to  surface 
protection  for  public  lands ;  continuing  review  of  practices,  regulations, 
and  procedures  of  BLM  and  related  Forest  Service  activities;  and 
review  of  forest  and  range  land  renewable  resources  practices. 

In  summary,  as  particularly  as  may  be  determined  at  the  present 
time,  the  Committee  on  Interior  and  Insular  Atfairs  during  the  94th 
Congress  may  probe  into  any  of  its  areas  of  jurisdiction  in  addition  to 
those  already  specifically  mentioned;  and  the  Committee  may  defer 
reviewing  some  of  the  jurisdictional  areas  now  being  considered  for 
oversight  activities.  In  any  event,  we  look  forward  to  cooperating  with 
the  Committee  on  Government  Operations  and  will  keep  you  advised 
as  our  schedule  is  revised  from  time  to  time. 
Sincerely  yours, 

James  A.  Haley,  Chairman. 
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COMMERCE 


House  of  Representatives, 
Committee  ox  Interstate  and  Foreign  Commerce, 

Washington.  B.C.,  February  28, 197o. 
Hon.  Jack  Brooks. 

Chairman,  Committee  on  Government  Operation*, 
House  of  Representative*, 
Washington,  B.C. 

Dear  Mr.  Chairman  :  As  required  by  Rule  X  2(c)  of  the  Rules  of 
the  House  of  Representatives,  appropriate  representatives  of  the 
Committee  on  Interstate  and  Foreign  Commerce  have  met  with  rep- 
resentatives of  the  Committee  on  Government  Operations  to  discuss 
the  oversight  plans  of  this  committee  during  this  Congress. 

The  Committee  will  be  organized  during  this  Congress  into  six  sub- 
committees, with  jurisdiction  as  follows : 
Subcommittee  on  Oversight  and  Investigations 

Responsibility  for  oversight  of  agencies,  departments  and  all  pro- 
grams within  the  jurisdiction  of  the  full  committee  and  to  conduct 
such  investigations  within  such  jurisdiction.  (Note:  This  subcommit- 
tee would  not  have  legislative  jurisdiction.  The  committees  legislative 
subcommittees  would  have  parallel  responsibility  to  conduct  oversight 
in  their  assigned  areas.  Rule  X(2)  (b)  of  House  Rules.) 

Subcommittee  on  Energy  and  Power 

Energy  matters  within  the  jurisdiction  of  the  full  committee,  and 
jurisdiction  over  all  petroleum,  natural  gas,  and  electrical  power 


Subcommittee  on  Communications 

Jurisdiction  over  interstate  and  foreign  communications,  including 
all  communications  by  satellite,  broadcast,  radio,  common  carriers, 
interstate  communication  bv  wire  and  such  jurisdiction  over  commu- 
nications and  media  as  in  tfxe  jurisdiction  of  the  full  committee. 
Subcommittee  on  Health  and  the  Environmen  t 

Responsibility  for  public  health  and  quarantine;  hospital  construc- 
tion; mental  health  and  research;  biomedical  programs  and  health 
protection  in  general,  including  medicaid  and  national  health  insur- 
ance ;  foods  and  drugs ;  and  drug  abuse. 

Jurisdiction  over  the  Clean  A.ir  Act,  the  State  Drinking  Water  Act, 
the  Radiation  for  Health  and  Safety  Act  and  environmental  protec- 
tion in  general. 
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Si'txo/nriutter  on  CvuMtmeV  Protection  and  Finance 

Jurisdiction  over  consumer  protection  in  general,  including  toxic 
substances,  product  safety,  motor  vehicle  safety,  securities  and  ex- 
changes (the  SEC),  the  regulation  of  trade  (the  FTC1),  and  insurance. 
Subcommittee  <>,>  Transportation  and  Commerce 

Jurisdiction  over  the  regulation  of  travel  and  tourism,  interstate 
and  foreign  commerce  generally,  and  all  matters  related  to  inland 
waterways,  railroads,  railroad  retirement,  railway  labor,  solid  waste 
disposal  ami  noise  pollution  control. 

OVERSIGHT    ACTIVITIES,    :'•!!)    CONGRESS 

In  all  programs  over  which  this  committee  has  jurisdiction  where 
it  is  appropriate  to  do  so.  the  committee  reports  legislation  to  the 
House  which  is  limited  in  duration,  generally  to  three  fiscal  years. 
This  means  that  at  least  once  every  three  years,  a  majority  of  the  pro- 
grams which  this  committee  is  responsible  for  are  reviewed,  during 
the  process  of  committee  consideration  of  legislation  extending  the 
duration  of  the  programs.  A  listing  of  these  programs  which  were 
within  the  committees  jurisdiction  during  the  03d  Congress  is  set 
forth  on  pages  110-238  of  the  committee's  activity  report  covering  the 
93d  Congress  (H.  Kept.  93-1668). 

In  addition,  each  of  the  legislative  subcommittees  of  the  full  com- 
mittee held  legislative  oversight  hearings  during  the  93d  Congress  on 
one  or  more  matters  within  the  subcommittee's  jurisdiction.  These 
hearings  covered  the  Florida  power  interruption,  the  Civil  Aeronau- 
tics Board.  Independent  truckers  and  the  energy  crisis,  the  National 
Highway  Traffic  Safety,  the  Motor  Vehicle  Information  and  Cost 
Savings  Act.  the  Consumer  Product  Safety  Commission,  HEW  health 
programs,  regional  medical  programs.  Community  Mental  Health 
Centers.  Public  Health  Service  Hospitals,  HEW  Reorganization, 
Clean  Air  Act.  Drug  Abuse.  Motor  Vehicle  standards  and  fuel  econ- 
omy, national  health  care  and  hospital  rate  regulation  and  Phase  IV 
cost  controls  for  health  services. 

In  addition  to  the  foregoing,  the  Special  Subcommittee  on  Investiga- 
tions conducted  numerous  oversight  activities,  summarized  on  pages 
61  through  70  of  the  committee's  activity  report  (II.  Rept.  93-1668), 
and  held  hearings  on  the  suhjects  of  freight  car  shortages,  SEC  trans- 
fer of  ITT  files  to  the  Justice  Department,  ex  parte  contacts  with  the 
SEC,  the  liquefied  natural  gas  tank  disaster  on  Staten  Island,  FTC 
practices  and  procedures,  FA  A  air  safety  oversight.  FAA-transporta- 
tion  of  hazardous  materials.  FAA-reliability  of  "drilled"  turbine  fan 
blades  on  CF-6  Kiiirine.  FAA-Ground  Proximity  Warning  System. 
FPC  re  GAO  report  on  gas  rates,  and  an  inquiry  into  CAB  rate  in- 
creases. 

I  anticipate  that  similar  activities  will  be  engaged  in  by  the  sub- 
committee during  the  coming  Congress. 

ANTICIPATED  OVERSIGIIT  ACTIVITIES,  D4TTI  CONGRESS 

During  the  94th  Congress.  I  anticipate  that  the  Committee,  either 
as  a  whole,  or  through  one  or  more  of  its  subcommittees,  will  engage 
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in  oversight  activities,  including  hearings,  involving  the  Federal  Com- 
munications Commission,  the  Office  of  Telecommunications  Policy,  the 
communications  satellite  program,  energy,  health  care  costs,  HEW 
health  programs,  both  generally  and  specifically,  the  administration 
of  laws  by  the  independent  regulatory  agencies  over  which  the  com- 
mittee has  jurisdiction,  problems  involving  the  railroads,  specifically 
the  problems  facing  the  Penn  Central  and  other  northeast  railroads, 
responsibility  for  public  health  and  health  protection  in  general,  in- 
cluding hospital  construction;  health  manpower;  nurse  training  and 
nursing  home  construction ;  mental  health ;  biomedical  research ;  drug 
abuse;  health  services  programs;  the  Public  Health  Service;  Indian 
health;  laws  affecting  foods,  drugs,  cosmetics  and  medical  devices; 
Medicaid;  and  national  health  insurance  proposals  funded  by  reve- 
nues not  derived  from  payroll  deductions,  among  others. 

It  is,  of  course,  too  early  in  this  session  of  the  Congress  to  determine 
what  the  full  scope  of  oversight  activities  by  the  committee  and  its 
subcommittees  will  be ;  however,  based  upon  the  past,  I  would  antic- 
ipate that  there  will  be  some  degree  of  oversight  activities  covering 
most  of  the  programs  within  the  committee's  jurisdiction,  and  over- 
sight hearings  covering  a  substantial  number  of  them. 
Sincerely  yours, 

Hakuey  O.  Staggers, 

Chairman. 
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House  of  Representatives, 
Committee  on  the  Judiciary, 
Washington,  D.C.,  March  3, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committer  on  Government  Operations,  House  of  Repre- 
sentatives, Washington,  D.C. 
Dear  Jack:  In  accordance  with  Clause  2(c)  of  Rule  X  of  the 
Rules  of  the  House  and  staff  discussions,  I  enclose  the  oversight  pro- 
gram of  the  Committee  on  the  Judiciary  for  the  94th  Congress.  Each 
subcommittee  will  exercise  oversight  responsibilities  within  its  area 
of  jurisdiction. 

The  enclosed  oversight  program  sets  forth  general  oversight  juris- 
diction of  each  subcommittee  as  well  as  specific  oversight  projects 
proposed  to  be  undertaken  by  each  subcommittee.  It  is  possible  that 
additional  oversight  projects  within  the  jurisdiction  of  the  Committee 
may  be  undertaken,  and  also  that  some  projects  may  be  deferred  or  not 
be  considered  during  the  94th  Congress. 
Kind  regards. 
Sincerely, 

Peter  W.  Rodino,  Jr.,  Chairman. 
Enclosure. 

subcommittee  on  immigration,  citizenship,  and  international  law 

The  Subcommittee  is  responsible  for  reviewing  on  a  continuing  basis 
the  administration  of  the  Immigration  and  Nationality  Act  by  the 
Departments  of  State,  Justice,  and  Labor.  Several  hearings  were  con- 
ducted during  the  92nd  and  93rd  Congresses  to  review  the  operation 
of  this  Act  including  pertinent  regulations,  operating  instructions, 
and  interpretations  of  that  Act  which  have  been  promulgated  by  the 
Executive  Branch.  The  Subcommittee  expects  to  continue  to  closelv 
monitor  the  implementation  of  this  basic  immigration  law  and  will 
attempt  to  identify  problems  which  have  developed  in  the  adminis- 
tration of  this  law. 

The  Subcommittee  also  intends  to  review  the  operation  of  the  Sub- 
merged Lands  Act  and  the  administration  of  the  Outer  Continental 
Shelf  Lands  Act  which  it  originally  processed  in  1953.  Preliminary 
hearings  were  conducted  in  the  second  session  of  the  93rd  Congress 
and  additional  hearings  are  planned  in  order  to  determine  whether 
amendments  to  this  basic  legislation  are  required.  The  administra- 
tion of  this  Act  has  become  particularly  important  in  view  of  the 
Administration's  plans  to  accelerate  the  leasing  of  oil  and  gas  on 
the  Outer  Continental  Shelf. 
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The  Subcommittee  plans  to  conduct  a  comprehensive  investigation 
of  the  foreign  student  issue.  This  includes  the  issuance  of  student 
visas  to  aliens,  the  role  of  the  educational  institutions  in  the  visa  is- 
suance process,  the  efforts  of  the  Immigration  and  Naturalization 
Service  to  monitor  the  foreign  student  program,  the  procedures  for 
granting  off -campus  and  summer  work  permission  to  foreign  students. 
the  approval  and  withdrawal  of  approval  for  educational  institutions 
receiving  foreign  students,  and  a  variety  of  other  problems  closely 
associated  with  the  foreign  student  program. 

The  Subcommittee  will  closely  scrutinize  the  various  refugee  prob- 
lems which  have  developed  around  the  world  and  will  conduct  hearings 
to  review  the  operation  of  the  Migration  and  Refugee  Assistance  Act 
of  1962.  In  addition,  the  Subcommittee  will  study  the  relationship 
between  the  1962  law  and  the  emergent  refugee  problems  which  have 
developed  in  various  countries  (i.e.,  Soviet  Jews,  Chilean  refugees, 
Haitian  refugees,  refugees  from  mainland  China).  In  this  regard,  the 
Subcommittee  intends  to  review  procedures  for  the  admission  of 
refugees  to  the  United  States  as  well  as  programs  which  have  been 
established  for  the  movement  and  resettlement  of  various  categories 
of  refugees. 

Another  topic  which  will  be  pursued  by  the  Subcommittee  in  this 
Congress  is  the  operation  of  the  labor  certification  program  prescribed 
in  Section  212(a)  (14)  of  the  Immigration  and  Nationality  Act.  The 
Subcommittee  is  expected  to  hold  several  hearings  to  consider  the 
utility  of  the  labor  certification  requirement  as  well  as  its  implementa- 
tion by  the  Departments  of  Justice  and  Labor. 

Finally,  the  Subcommittee  will  study  the  provisions  of  the  Immi- 
gration and  Nationality  Act  relating  to  the  exclusion  and  deportation 
of  those  who  will  or  have  become  public  charges. 

The  aforementioned  oversight  projects  are  those  which  the  Sub- 
committee intends  to  carry  out  at  the  present  time. 

SUBCOMMITTEE  ON  ADMINISTRATIVE  LAW  AND-  GOVERNMENTAL  RELATIONS 

The  oversight  jurisdiction  of  the  Subcommittee  over  the  Depart- 
ment of  Justice  relates  primarily  to  the  Civil  Division,  with  emphasis 
on  those  actvities  involving  torts,  claims  by  and  against  the  Govern- 
ment, and  contracts.  Justice  Department  oversight  also  includes  lands 
division  activity,  with  reference  to  the  recently-enacted  law  providing 
jurisdiction  in  the  Federal  courts  to  try  land  titles  involving  the 
United  States.  The  claims  jurisdiction  of  the  Subcommittee  requires 
oversight  over  the  administration  of  the  claims  statutes  by  executive 
departments  and  agencies,  including  the  military  departments  and 
their  operations  both  in  this  country  and  abroad.  Also  included  in  its 
responsibility  is  oversight  of  the  Secret  Service  relating  to  protection 
of  the  President  and  other  persons.  Incident  to  its  iurisdiction  over 
legislation  relating  to  the  termination  of  National  Emergencies,  the 
Subcommittee  has  oversight  over  the  administration  of  operation  of 
the  various  agencies  of  Government  under  emergency  statutes,  with 
particular  reference  to  the  Department  of  Defense  and  the  Treasury 
Department.  Under  its  jurisdiction  over  administrative  law,  the  Sub- 
committee will  be  required  to  exercise  jurisdiction  over  those  depart- 


413 


59 

ments  and  agencies  operating  under  the  Administrative,  Procedure 
Act  and  other  functions  falling  within  the  area  of  administrative  kiw. 
The  Subcommittee  contemplates  a  long-term  examination  of  the 
process  of  adjudication  and  decision-making  in  Government  in  the 
area  of  administrative  law  and  practice.  This  will  include  an  exami- 
nation of  the  current  application  of  the  Administrative  Procedure 
Act  but  will  extend  beyond  that  to  a  basic  analysis  of  adjudication 
and  settlement.  Under  multiple  laws,  the  agencies  and  departments  of 
Government  settle  claims  arising  from  Government  activity.  This 
too  will  be  a  continuing  study  with  the  immediate  examination  relat- 
ing to  the  Tort  Claims  Act.  The  emergency  powers  operations  of 
Government  are  of  immediate  concern  to  the  subcommittee  and  will 
be  the  subject  of  examination  in  the  near  future. 

SUBCOMMITTEE    ON    COURTS,   CIVIL   LIBERTIES,   AND   THE   ADMINISTRATION" 
OF   JUSTICE 

Pursuant  to  jurisdiction  assigned  by  Rule  VI  of  the  Committee's 
Rules  of  Procedure,  the  Subcommittee  plans  to  exercise  general  over- 
sight in  the  94th  Congress  with  regard  to  the  Federal  Bureau  of  Pris- 
ons, the  Patent  Office,  the  Copyright  Office,  the  Federal  Court  System, 
the  Presidential  Clemency  Board,  and  Government  agencies  conduct- 
ing surveillance  of  American  citizens.  Specific  oversight  projects  now 
being  planned  or  already  initiated  include  Bureau  of  Prisons  Con- 
struction Plans  and  Behavior  Modification  Program;  Federal  Pris- 
on Industries;  the  Presidential  Clemency  Board;  electronic  surveil- 
lance by  Federal  agencies  and  the  Justice  Department  pilot  project 
on  pretrial  diversion. 

Specific  legislative  measures  to  be  considered  will  include  a  variety 
of  proposed  improvements  in  the  corrections  systems,  a  copyright  law 
revision,  and  a  variety  of  proposals  to  improve  the  Federal  Court 
System. 

It  must,  of  course,  be  understood  that  the  foregoing  is  a  highly  ten- 
tative estimate,  subject  to  change  with  changes  in  the  legislative  scene. 
This  may  include  removal  of  subjects  from  consideration  as  well  as 
the  addition  of  further  areas  of  study. 

SUBCOMMITTEE   OX   CIVIL   AND   CONSTITUTIONAL   RIGHTS 

The  Subcommittee  on  Civil  and  Constitutional  Rights  has  been 
assigned  oversight  responsibilities  for  the  Federal  Bureau  of  Inves- 
tigation, the  Civil  Rights  Division  of  the  Department  of  Justice  and 
the  Community  Relations  Service. 

The  Subcommittee  has  undertaken  and  is  presently  pursuing  the 
following  oversight  projects : 

1.  The  Operations  of  the  FBI 

Hearings,  staff  and  GAO  studies  are  underway  to  determine  the 
economy,  efficiency,  and  effectiveness  of  the  FBI.'  The  areas  of  con- 
centration at  the  moment  are: 

(a)  the  coimterintelligenee  programs; 

(b)  information  gathering  and  record  keeping  activities; 
(r)  statistical  reporting  systems  and  procedures; 

(d)  the  authority  underlying  all  FBI  activities. 
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£.  fivil  Rights  Oversight 

(a)  Staff  studies  and  hearings  are  underway  relative  to  the 
enforcement  of  the  civil  rights  provisions  of  the  act  commonly 
known  as  General  Revenue  Sharing. 

(b)  Oversight  is  continuing  in  the  areas  of  Federal  employment 
problems  of  Spanish-speaking  Americans  and  particularly  in 
the  area  of  enforcement  of  the  Voting  Rights  Act  by  the  Civil 
Rights  Division  of  the  Department  of  Justice. 

(c)  Oversight  is  also  continuing  in  the  area  of  equal  housing 
opportunities  and  in  the  enforcement  of  Title  VI  of  the  Civfl 
Rights  Act  of  1964  in  programs  receiving  medicare  and  medicaid 
funds. 

SUBCOMMITTEE   ON   MONOPOLIES  AND   COMMERCIAL   LAW 

The  Monopolies  and  Commercial  Law  Subcommittee  has  general 
oversight  responsibilities  in  the  field  of  enforcement  of  the  antitrust 
laws.  This  includes  the  activities  of  the  antitrust  division  of  the  De- 
partment of  Justice  and  the  Federal  Trade  Commission,  as  well  as  a 
variety  of  other  agencies  responsible  for  the  administration  of  pro- 
grams and  policies  within  the  framework  of  antitrust  enforcement  or 
embodying  exemptions  from  the  antitrust  laws. 

The  Subcommittee  envisions  major  hearings  this  year  on  the  en- 
forcement of  the  antitrust  laws  in  the  general  field  of  development  and 
distribution  of  energy  resources.  These  hearings  will  focus  specifically 
on  the  problems  posed  by  the  emergence  of  so-called  "energy  conglom- 
erates". The  Subcommittee  further  anticipates  review  of  the  applica- 
tion of  the  new  criminal  provisions  of  the  antitrust  laws  enacted  by  the 
previous  Congress,  a  study  of  the  need  for  expansion  of  the  power  of 
the  antitrust  division  under  the  Antitrust  Civil  Process  Act,  and  a 
general  review  of  the  interplay  between  private  and  governmental  (in- 
-cluding  both  state  and  federal)  enforcement  of  the  antitrust  laws. 

SUBCOMMITTEE   ON    CRIME 

The  Subcommittee  on  Crime  has  oversight  jurisdiction  over  the  law 
Enforcement  Assistance  Administration  and  the  Drug  Enforcement 
Administration.  The  Subcommittee  will  exercise  its  responsibility  with 
respect  to  LEAA  beginning  in  midyear  1975.  The  staff  is  presently  in- 
vestigating the  following  areas  for  LEAA  oversight  hearings :  LEAA 
and  its  response  to  the  problems  of  state  courts,  law  enforcement  and 
corrections ;  effectiveness  of  block  and  discretionary  grant  programs ; 
long-term  impact  of  Law  Enforcement  Assistance  grants ;  and  evaluat- 
ing the  results  of  LEAA  programs. 

With  respect  to  the  Drug  Enforcement  Administration,  the  Sub- 
committee staff  is  reviewing  the  following:  areas  for  oversight  hear- 
ings :  jurisdictional  problems  in  the  Drug  Enforcement  between  DEA 
and  Customs :  and  effectiveness  of  DEA  overseas  operations. 

In  addition  to  the  above  areas,  the  Subcommittee  may  consider  other 
possible  subjects  for  oversight  hearings.  The  final  decision  awaits  re- 
view by  the  Subcommittee. 
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SUBCOMMITTEE   ON   CRIMINAL  JUSTICE 

The  Criminal  Justice  Subcommittee  has  been  assigned  oversight 
responsibilities  for  the  activities  of  the  Justice  Department's  Criminal 
Division.  The  Subcommittee  has  asked  Acting  Assistant  Attorney  Gen- 
eral John  C.  Keeney  to  prepare  a  report  on  the  organization  and  oper- 
ation of  the  Criminal  Division. 

The  Subcommittee  is  meeting  on  March  12  to  review  this  report.  At 
that  time,  the  Subcommittee  will  decide  what  areas  of  Criminal  Divi- 
sion activities  to  look  into.  There  has  been  interest  expressed  in  review- 
ing the  exercise  of  prosecutorial  discretion  in  conspiracy  prosecutions 
and  in  reviewing  the  use  of  the  witness  immunity  provisions  of  Title 
18.  Oversight  hearings  in  both  of  these  areas  would  be  carried  out  in 
conjunction  with  the  Subcommittee's  work  on  criminal  code  reform 
legislation. 

The  Criminal  Justice  Subcommittee  will  have  to  maintain  close  liai- 
son with  the  Subcommittee  on  Courts,  Civil  Liberties,  and  the  Admin- 
istration of  Justice,  which  has  oversight  responsibility  for  the  activi- 
ties of  U.S.  Attorneys.  Prosecutions  under  many  of  the  statutes  that 
the  Criminal  Division  has  jurisdiction  over  are  carried  out  by  U.S. 
Attorneys.  Thus,  the  Criminal  Justice  Subcommittee's  oversight  activi- 
ties may  overlap  those  of  the  Subcommittee  on  Courts,  Civil  Liberties, 
and  the  Administration  of  Justice. 
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U.S.  House  of  Representatives, 
Committee  ox  Merchant  Marine  and  Fisheries. 

Washington.  D.C.  February  28.  197'j. 
Hon.  Jack  Brooks. 

Chairman,  Hovse  Government  Operations  Committee.  2157  Rayhvrn 
House  Office  Building,  Washington,  D.C. 

Dear  Mr.  Chairman  :  Tliis  is  in  reply  to  your  letter  of  February  13, 
1975,  in  which  you  requested  a  list  of  oversight  activities  contem- 
plated by  the  Committee  on  Merchant  Marine  and  Fisheries  during 
the  present  Congress.  The  information  is  needed  in  order  to  enable 
you  to  fulfill  your  responsibilities  under  Clause  2(c)  of  Rule  X  of 
the  Rules  of  the  House. 

In  adopting  its  Committee  Rules  for  the  91th  Congress,  the  Com- 
mittee on  Merchant  Marine  and  Fisheries  has  specifically  incorporated 
by  reference  the  oversight  provisions  of  the  House  Rules.  In  per- 
forming its  oversight  functions,  the  Committee  has  elected  to  operate 
through  its  regular  subcommittees  without  the  establishment  of  a 
Subcommittee  on  Oversight.  Each  Subcommittee  has  subsequently 
developed  plans  for  performing  oversight  in  its  own  area  of  jurisdic- 
tion. 

I  am  sure  that  you  will  realize  that  our  present  oversight  plans, 
while  generally  complete  as  we  see  them  at  this  time,  will  necessarily 
be  subject  to  change  with  changing  circumstances  during  the  course 
of  the  next  two  years.  Furthermore,  changes  in  priority  may  cause 
adjustments  in  future  activities.  Subject  to  that  caveat,  the  Commit- 
tee oversight  plans  are  reasonably  firm  and  are  listed  in  the  attached 
material,  divided  generally  into  areas  of  Subcommittee  activity.  I 
should  point  out  in  that  connection  that,  in  addition  to  the  specific 
listings,  two  of  the  Subcommittees  perform  jreneral  agency  oversight 
in  the  consideration  of  annual  authorization  bills  involving  the  Mari- 
time Administration  and  the  Coast  Guard. 

I  trust  that  the  material  as  presented  is  consistent  with  your  require- 
ments. 

Sincerely, 

LeoxorK.  (Mrs.  Johx  B.)  Sullivan. 

Chairman. 

a.  subcommittee  ox  coast  guard  and  xavigattox 

1.  Marine  safety  generally,  including:  measures  relating:  to  commer- 
cial vessel  construction,  loadlinos.  navigational  rules  and  personnel 
licenses,  and  recreational  boating  construction  standards  and  opera- 
tions. 
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2.  Marine  traffic  management,  and  particularly  vessel  traffic  systems 
in  U.S.  navigable  waters. 

3.  Pollution  control  measures,  including  specifically  tank  vessel 
construction  and  operation. 

4.  Marine  law  enforcement  generally,  including  ant i -smuggling, 
offshore  fisheries  laws  and  treaties,  and  ocean  dumping,  as  well  as  the 
implementation  of  the  Deep  Water  Port  Act  of  1974  as  it  relates  to 
safety  and  anti-pollution  measures  in  particular. 

5.  Coast  Guard  procurement  practices. 

B.    SUBCOMMITTEE    ON    FISHERIES    AND    WILDLIFE    CONSERVATION'    AND  THE 
ENVIRONMENT 

1.  Commercial  fishing  generally,  arid  the  implementation  of  fish- 
ing management  practices. 

2.  National  wildlife  refuge  administration. 

3.  Implementation  of  fishing  treaties  and  conventions,  including 
agreements  on  coastal  fisheries,  high  seas  salmon  fisheries,  and  shrimp 
fisheries  in  foreign  waters. 

4.  Implementation  of  various  laws,  including — 

National  Environmental  Policy  Act  of  1969 
Marine  Mammal  Protection  Act  of  1972 
Endangered  Species  Act  of  1973 
Migratory*  Bird  Acts 
Fish  and  Wildlife  Coordination  Act 
Wetlands  Acquisition  Act  of  1961 
Fishermen's  Protection  Act  of  1967 
and  various  laws  related  to  conservation  programs. 

C.    SUBCOMMITTEE    ON    MERCHANT    MARINE 

1.  Maritime  Administration  generally,  and  the  Federal  Maritime 
Commission. 

■2.  Maritime  education  and  training. 

3.  Maritime  research  and  development. 

4.  Maintenance  of  the  National  Defense  Reserve  Fleet. 

5.  Financial  responsibility  requirements  for  foreign  vessels  operat- 
ing in  United  States  waters. 

6.  Implementation  of  the  Merchant  Marine  Act,  1936,  as  amended 
relating  particularly  to  construction  subsidy  funds,  negotiated  con- 
tract of  subsidized  ship  construction,  operating  subsidy  funds,  the  loan 
guarantee  program,  and  the  war  risk  insurance  program. 

7.  Implementation  of  the  Shipping  Act,  1916,  as  amended,  relating 
particularly  to  conference  activities,  dual  rate  contract  activities  the 
licensing  and  regulation  of  ocean  freight  forwarders. 

D.    SUBCOMMITTEE   ON    OCEANOGRAPHY 

1.  Marine  resource  development,  including  Outer  Continental  Shelf 
resources  and  the  impact  of  their  exploitation  on  the  marine 
environment. 

2.  Oceanographic  research  generally,  including  adequate  safetv 
measures  for  oceanographic  research  vessels  and  platforms. 
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3.  National  Advisory  Committee  on  Oceans  and  Atmosphere,  and 
the  implementation  of  its  recommendations. 

4.  Implementation  of  the  Marine  Protection.  Research,  and  Sanctu- 
aries Act  of  1972,  particularly  in  relation  to  alternatives  for  solid 
waste  disposal,  in  ocean  areas. 

5.  Implementation  of  the  Coastal  Zone  Management  Act  of  1972, 
particularly  in  its  relation  to  activities  on  the  O • -ter  Continental  Shelf 
and  the  impact  of  those  activities  on  coastal  states. 

6.  Implementation  of  the  Sea  Grant  College  and  Program  Act 
of  1966. 

E.    SUBCOMMITTEE    ON    THE    PANAMA    CANAL 

1.  Personnel  policies  of  the  Panama  Canal  Company  and  the  Canal 
Zone  Government. 

2.  Civil  operations  of  the  Canal  Zone  Government. 

3.  Tolls,  toll  structures,  and  financial  management  of  the  Canal. 

4.  The  relationship  of  the  Panama  Canal  to  world  shipping  and  to 
national  security. 

5.  Implementation  of  the  Coastal  Zone  Management  Act  of  1972. 

6.  Labor-management  relations  in  the  Canal  Zone. 

7.  Implications  of  treaty  negotiations  on  future  Canal  Zone  oper- 
ations. 
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U.S.  House  of  Representatives, 
Committee  on  Post  Office  and  Civil  Service, 

Washington,  D.C.,  February  25, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  House  of  Repre- 
sentatives, Washington,  D.C. 
Dear  Mr.  Chairman  :  As  requested  by  your  letter  of  February  13, 
1975,  there  is  attached  the  report  of  the  Post  Office  and  Civil  Service 
Committee  listing  the  oversight  activities  contemplated  by  the  Com- 
mittee during  the  94th  Congress. 
With  kindest  regards,  I  am 
Sincerely  yours, 

David  N.  Henderson,  Chairman. 
Enclosure. 

LEGISLATIVE  OVERSIGHT  ACTIVITIES  PROPOSED  FOR  THE  94TH  CONGRESS 

Committee  Organization 

Rule  20  of  the  Rules  of  the  Post  Office  and  Civil  Service  Commit- 
tee establishes  six  subcommittees,  each  with  both  legislative  and  over- 
sight responsibilities  for  the  matters  under  their  respective  jurisdic- 
tions, as  provided  under  Committee  Rule  21.  In  addition,  Rule  20 
authorizes  the  Chairman,  or  the  Committee  by  a  majority  vote,  to  es- 
tablish special  or  ad  hoc  investigative  subcommittees  and  assign  to 
them  such  jurisdiction  as  the  Chairman  of  the  Committee  deems  neces- 
sary. No  special  or  ad  hoc  investigative  subcommittee  has  been  estab- 
lished at  this  time. 

Scope  of  Oversight  Responsibilities 

The  Committee  has  continuing  oversight  responsibilities,  under 
House  Rule  X(o) ,  over  the  United  States  Census,  United  States  Postal 
Service,  Postal  Rate  Commissions^  and  the  United  States  Civil  Service 
Commission,  and  the  programs  administered  by  the  agencies  involved. 
In  the  case  of  the  Civil  Service  Commission,  this  includes  the  Federal 
Civil  Service  and  the  competitive  Civil  Service  generally,  Federal  em- 
ployee pay  and  fringe  benefit  matters,  including  Civil  Service  retire- 
ment, life  and  health  insurance  programs;  employee  political  rights 
(Hatch  Act  matters) ;  and  the  intergovernmental  Personnel  Programs. 
(65) 
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Orcrxight  Program  for  the  9^th  Congress 

The  Committee,  acting  through  the  six  subcommittees,  has  adopted 
an.  oveisight  program  on  the  subject  matters  outlined  below : 
Manpower  and  Ciril  Service  Subcommittee 

1.  Violations  and  abuses  of  the  Federal  Civil  Service  merit  prin- 
ciples, including  examination  and_recruitment  for  high  level  positions: 
relationship  between  career  Civil  Service  and  political  appointees: 
review  and  evaluation  of  agency  controls  to  insure  compliance  with 
the  merit  system:  and  effectiveness  of  the  controls  for  determining 
whether  particular  positions  should  be  exempted  from  the  competitive 
service. 

2.  The  system  of  examining  and  issuance  of  certificates  of  eligibility 
for  competitive  Civil  Service  positions  and  the  use  of  the  three  top 
eligibles  for  such  certificates  of  eligibility. 

3.  The  ranking  of  positions  under  the  Executive  Salary  Schedule 
(5  U.S.C.  5311-5317). 

-k  Personnel  and  pay  systems  of  the  Panama  Canal  Company  and 
their  l-elationship  to  personnel  and  pay  systems  for  employees  of  other 
executive  departments  in  the  Canal  Zone. 

5.  The  impact  of  personnel  systems  and  pay  policies  on  employees 
of  the  scheduled  disestablishment  of  Headquarters  Pacific  Air  Forces 
and  U.S.  Army  Headquarters  Pacific. 

G.  Civilian  manpower  utilization  studies,  including  the  review  sys- 
tem for  determining  "in  house"  versus  technical  support  service  con- 
tract requirements;  the  use  of  statutory  personnel  ceilings  versus 
effective  dollar  economies;  and  the  proper  use  of  civilians  in  positions 
occupied  in  the  Department  of  Defense  by  military  personnel. 
Postal  Service  Subcommittee 

1.  The  financial  status  of  the  United  States  Postal  Service,  including 
proposals  for  increased  appropriations  for  subsidies ;  the  use  of  bond 
revenues  for  operating  expenses;  and  the  allocation  of  cost  between 
the  various  postal  services. 

2.  Review  of  the  function  and  operation  of  the  Board  of  Governors 
of  the  Postal  Service. 

3.  Review  of  the  organization  and  operation  of  the  United  States 
Postal  Rate  Commission,  including  the  hearing  procedures;  appoint- 
ment and  qualifications  of  the  Commissioners;  and  the  authority  of 
the  Commission  to  prescribe  the  procedure  and  information  to  be 
furnished  by  the  Postal  Service  in  a  postal  rate  or  classification 
hearing. 

4.  A  review  of  the  application  of  the  Private  Express  Statutes. 

5.  Review  of  the  standards  and  methods  of  delivery  of  the  United 
States  mail  and  the  handling  of  military  mail;  and  review  of  postal- 
related  services,  including  the  sale  of  envelopes  and  packing  materials. 
Postal  Facilities.  Mail,  and  Labor  Management  Subcommittee 

1.  Review  of  the  labor-management  issues  in  the  Postal  Service. 
including  the  right  to  strike,  uniformity  in  labor  relations,  an  arbitra- 
tion board  system  to  settle  disputes  with  postal  supervisors,  and  safety 
programs  for  postal  employees. 

2.  Review  of  the  Letter  Carrier  Route  Evaluation  Svstem  and  the 
system  for  central  mark-up  of  mail. 

3.  A  review  of  the  postal  facilities  improvement  and  mechanization 
program  and  the  real  estate  acquisition  policies. 
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4.  A  review  of  the  mailability  of  particular  mail  matter,  including 
obscene  material,  lottery  information,  and  unsolicited  drugs. 

...  A  review  of  tin-  program  for  the  standardization  of  envelopes, 
and  size  and  weight  limits  for  parcels. 

(i.  The  impact  on  the  United  States  mails  of  the  electronic  trans- 
mission of  information. 

7.    Review    of   the   program    for   acquisition   of   data   processing 
equipment. 
Retirement  and  Employee  Benefits  Subcommitfi  e 

1.  Review  of  the  financial  condition  of  the  Civil  Service  Retirement 
Fund  and  the  procedures  of  investing  the  current  surplus  funds. 

2.  A  review  to  compare  the  Federal  Employees'  Life  Insurance  Pro- 
gram benefits  and  costs  with  comparable  program  benefits  and  costs  in 
private  industry. 

3.  A  review  of  the  pay  comparability  system  used  for  adjusting  the 
pay  of  Federal  employees  under  the  statutory  salary  systems. 

4.  A  review  of  certain  matters  related  to  the  Health  Benefits  Pro- 
gram, including  inequities  in  benefits  and  costs  arising  because  of 
certain  restrictions  in  State  laws;  the  advisability  of  extending  the 
program  to  include  coverage  by  additional  insurance  companies  on  the 
basis  of  competitive  bidding;  a  review  of  the  administration  of  the 
program  by  the  Civil  Service  Commission;  and  the  feasiblity  of  con- 
verting the  program  to  a  Federally  owned  and  operated  program. 

Employee  Political  Rights  and  Intergovernmental  Programs  Subcom- 
mittee 

1.  A  review  of  the  political  rights  of  Federal  employees  under  the 
Hatch  Act  and  the  administration  of  that  Act  by  the  Civil  Service 
Commission. 

2.  Review  of  the  administration  and  the  operation  of  the  Intergov- 
ernmental Personnel  Program,  including  the  adequacy  of  the  present 
limitations  on  the  grant-in-aid  authority;  the  exchange  of  Federal 
and  State  employees;  and  Equal  Employment  Opportunity  in  local 
personnel  systems. 

Census  and  Population  Subcommittee 

1.  A  review  of  the  plans  of  the  Bureau  of  Census  for  the  1980  census 
program,  including  the  protection  and  the  confidentiality  of  the  census 
information. 

2.  A  review  of  the  statistics  collection  program  of  the  Bureau  of 
Census,  including  the  collection  of  unemployment  statistics. 
Future  Oversight  Review  Programs 

The  Committee  has  the  policy  of  reviewing  recommendations  in- 
volving the  agencies  under  our  oversight  jurisdiction  included  in 
General  Accounting  Office  reports.  Also,  the  Committee  is  continuallv 
interested  in  current  reports  issued  by  the  Civil  Service  Commission 
on  the  operations  of  the  Civil  Service  by  agencies  of  the  Executive 
Branch  and  of  operations  under  the  retirement  and  life  insurance  pro- 
grams. Any  one  of  these  matters  may  result  in  additional  programs 
that  would  be  proper  for  a  legislative  review  by  one  of  our  subcom- 
mittees at  am   ' ' 


ny  time  during  the  period  covered  by  this  report. 
Also,  the  Committee  will  continue  to  exercise  its  legislativ 


e  over- 


sight function  with  regard  to  :ny  matter  under  the  Committee's  juris- 
diction involving  circumstances  justifying  a  review  bv  the  Committee 
during  the  94th  Congress. 
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COMMITTEE  ON  PUBLIC  WORKS  AND 
TRANSPORTATION 


House  of  Representatives, 
Committee  on  Public  Works  and  Transportation, 

Washington,  D.C.,  March  3, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations, 
House  of  Representatives^  Washington,  D.C. 

Dear  Jack  :  Enclosed  is  a  memorandum  of  information  in  response 
to  your  recent  request  for  the  Committee  on  Public  Works  and  Trans- 
portation's oversight  responsibilities  and  activities. 

I  trust  this  information  will  be  helpful  to  you  and  that  you  will 
contact  me  if  I  can  supply  you  with  any  additional  information  or  be 
of  assistance  to  you  in  any  way. 

With  every  good  wish  and  warm  personal  regards, 
Sincerely, 

Robert  E.  Jones,  Chairman. 
Enclosure. 

MEMORANDUM 

Subject:  Oversight  Activities  of  the  Committee  on  Public  WorKs 
and  Transportation,  94th  Congress. 

I.  Mechanics  of  Oversight. — The  Committee  has  had,  since  Sep- 
tember 15,  1959,  a  fully  functional  Subcommittee  on  Investigations 
and  Review.  This  subcommittee  is  comprised  of  22  members  and  has 
a  staff  of  19.  This  subcommittee  and  the  appropriate  subcommittee 
with  legislative  authority  cooperatively  review  and  study,  on  a  con- 
tinuing basis,  the  application,  administration,  execution,  and  effec- 
tiveness of  those  laws,  or  parts  of  laws,  within  the  Committee's  juris- 
diction. The  oversight  may  be  carried  out  by  the  Subcommittee  on 
Investigations  and  Review,  by  the  legislative  subcommittee  or  both. 
In  similar  fashion,  the  Federal  agencies  and  entities  having  respon- 
sibilities for  the  administration  and  execution  of  these  laws  are  re- 
viewed in  order  to  determine  whether  such  laws  and  the  programs 
thereunder  are  being  implemented  and  carried  out  in  accordance  with 
the  intent  of  Congress  and  whether  such  programs  should  be  con- 
tinued, curtailed,  modified  or  eliminated. 

II.  General  Outline  of  Oversight  Responsibilities. — The  Commit- 
tee has  oversight  responsibilities  in  the  following  general  program 
areas:  National  transportation  policy;  civil  aviation,  including  Air- 
port Development  and  the  activities  of  the  Civil  Aeronautics  Board, 
the  Federal  Aviation  Administration,  and  the  National  Transporta- 
tion Safety  Board ;  surface  transportation,  including  highways,  mass 
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transportation,  highway  motor  carrier  and  motor  vehicle  safety,  and 
the  activities  of  the  Interstate  Commerce  Commission;  development 
of  the  nation's  water  resources,  including  water  pollution  control, 
deep  water  ports,  the  activities  of  the  U.S.  Army  Corps  of  Engineers, 
the  St.  Lawrence  Seaway  Development  Corporation  and  the  Ten- 
nessee Valley  Authority ;  the  Economic  Development  Administration 
and  related  economic  development  programs,  including  Appalachian 
regional  development ;  watershed  program  of  the  Soil  Conservation 
Service;  public  buildings  and  grounds;  uniform  relocation  assistance; 
and  disaster  relief. 

III.  Likely  Areas  of  Concentration  in  the  9^th  Congress. — Imple- 
mentation of  Federal  Water  Pollution  Control  Program,  highway 
and  air  safety,  transportation  regulatory  reform,  construction  over- 
runs, water  resources  project  evaluation,  disaster  relief,  relocation  as- 
sistance program,  public  mass  transportation,  and  p reconstruction 
delays. 

IV.  It  should  be  noted  that  the  above  statements  are  general  in 
nature  and  are  intended  to  be  neither  all  inclusive  nor  unchangeable. 
It  is  possible,  if  not  likely,  that  the  course  of  events  during  the  next 
two  years  will  cause  the  Committee  to  apply  its  oversight  capabilities 
in  areas  not  specifically  foreseen  at  this  time  and  to  place  less  em- 
phasis than  is  now  anticipated  on  areas  mentioned  herein. 
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U.S.  House  of  Representatives, 

Committee  on  Rules, 
Washington,  D.C.,  February  28, 1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  U.S.  House  of 
Representatives,  Washington,  D.C. 

Dear  Mr.  Chairman:  Chairman  Madden  is  happy  to  submit  the 
following  information  in  line  with  your  request  for  certain  plans  for 
operation  of  the  House  Committee  on  Rules  for  1975. 

Although  committee  rules  have  been  adopted  and  the  House  Rules 
Committee  is  functioning,  the  issue  of  additional  staff  has  been  post- 
poned and  the  Committee  has  not  decided  to  operate  with  subcommit- 
tees to  date.  The  Rules  Committee  is  exempt  from  the  requirement  to 
form  subcommittees.  However,  the  jurisdiction  and  responsibility  of 
the  Rules  Committee  has  been  broadened  considerably  by  the  new 
changes  in  the  House  Rules. 

One  of  the  most  important  new  assignments  of  the  House  Rules 
Committee  is  its  general  oversight  responsibilities  which  it  has  never 
had  before.  "Each  standing  committee  .  .  .  shall  review  and  study,  on 
a  continuing  basis,  the  application,  administration,  execution,  and 
effectiveness  of  those  laws,  or  parts  of  laws,  the  subject  matter  of  which 
is  within  the  jurisdiction  of  that  committee,  and  the  organization  and 
operation  of  the  Federal  agencies  ...  In  addition,  each  such  commit- 
tee shall  review  and  study  any  conditions  or  circumstances  which  may 
indicate  the  necessity  or  desirability  of  enacting  new  or  additional 
legislation  .  .  ." 

This  oversight  function  clearly  includes  the  Legislative  Reorganiza- 
tion Act  of  1970  and  the  Congressional  Budget  and  Impoundment 
Control  Act  of  1974.  which  this  Committee  wrote.  It  also  includes 
oversight  of  Boiling  Resolution.  H.  Res.  988. 

(a)  The  new  Rules  give  the  House  Committee  on  Rules  the  respon- 
sibility for  emergency  waivers  of  the  required  reporting  date  for 
bills  and  resolutions  authorizing  new  budget  authority.  This  means 
that  the  Rules  Committee  should  follow  the  new  legislative  Budget 
Committee  in  its  progress  very  closely  on  a  continuous  basis  in  order 
to  be  in  a  position  of  expediting  its  operations  in  case  of  difficulties 
with  the  other  legislative  committees,  the  other  body,  or  in  other  pos- 
sible situations  which  may  arise  to  cause  the  House  Budget  Commit- 
tee to  be  inoperative. 

(b)  The  Joint  Committee  on  Congressional  Operations  is  assigned 
the  responsibility  of  conducting  a  continuous  study  of  the  jurisdiction 
of  the  various  standing  committees  of  the  House  and  must  report 
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periodically  its  results  to  the  House  Committee  on  Rules.  It  is  in- 
cumbent upon  the  House  Committee  on  Rules  to  consider  these  studies 
and  recommendations  relative  to  updating  the  jurisdictions  of  the 
various  standing  committees.  Evidently,  there  is  a  continuing  need 
in  this  area.  For  example,  H.R.  2633,  a  bill  to  increase  domestic  energy 
supplies  and  availability,  was  referred  to  four  different  committees 
under  the  new  rules.  This  will  not  simplify  a  hearing  before  the  Rules 
Committee  if  or  when  the  Armed-Services,  Interstate  and  Foreign 
Commerce,  Ways  and  Means,  and  the  Banking,  Currency,  and  Hous- 
ing Committees  testify  on  this  measure.  Some  legislation  has  been 
referred  to  five  committees. 

(c)  These  Rules  changes  and  specific  assignments,  along  with  the 
nature  tit  the  overall  across-the-board  consideration  for  rules  on  every 
conceivable  subject  from  twenty  legislative  committees,  seem  to  put 
the  House  Rules  Committee  in  a  position  of  monitoring  or  perform- 
ing a  broad  oversight  function — especially  as  it  relates  to  the  legisla- 
tive budget  operations,  the  Joint  Committee  on  Congressional  Opera- 
tions, and  perhaps  other  pressing  issues  that  are  disturbing  our  coun- 
try. This  latter  responsibility  seems  to  be  tied  in  with  the  authority  of 
the  House  Rules  Committee  to  discharge  other  committees  on  issues 
of  national  or  international  consequences  which  are  bogged  down 
somewhere  along  the  legislative  process. 

The  Chairman  and  the  Committee  staff  are  happy  to  cooperate  with 
you  in  this  undertaking  and  hope  that  the  above  information  is  ap- 
propriate to  your  request. 

Rest  personal  regards. 
Sincerely, 

L.  C.  Battle.  Staff  Director. 
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U.S.  House  or  Representatives, 
Committee  on   Science  and  Technology, 

Washington,  D.C.,  February  28,  1975. 
Hon.   Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  House  of  Repre- 
sentatives, Washington,  D.G. 

Dear  Mr.  Chairman  :  In  response  to  your  letter  of  February  13, 1 
am  reporting  the  oversight  activities  contemplated  by  the  Committee 
on  Science  and  Technology  during  the  94th  Congress. 

Oversight  is  a  responsibility  of  the  Legislative  branch  which  the 
Committee  on  Science  and  Technology  takes  very  seriously.  A  contin- 
uing review  of  the  administration,  execution  and  effectiveness  of  the 
federal  agencies  and  laws  enacted  for  which  this  Committee  has  been 
given  responsibility  is  carried  out  by  the  Subcommittees  when  there 
are  well  denned  areas  of  jurisdiction.  Where  there  is  overlap  of  juris- 
dictions, oversight  is  retained  for  full  Committee  action. 

General  oversight  activities  of  the  Committee  include  a  continuing 
review  of  the  National  Aeronautics  and  Space  Administration,  the 
National  Science  Foundation,  the  National  Bureau  of  Standards,  the 
non-nuclear  portion  of  the  Energy  Research  and  Development  Admin- 
istration and  the  Research  and  Development  programs  of  the  Environ- 
mental Protection  Agency.  In  addition,  the  Committee  Reform 
Amendment  of  1974  gave  the  Committee  on  Science  and  Technology 
the  responsibility  of  Special  Oversight  over  all  non-military  research 
and  development.  This  is  a  comprehensive  responsibility  which  the 
Committee  intends  to  exercise  aggressively  and  yet  wisely. 

Specific  oversight  reviews  and  inquiries  are  planned  in  the  follow- 
ing areas  of  primary  and  special  oversight  jurisdictions. 


International  cooperative  programs  in  Space. 

Need  for  international  agreements  and  conventions  to  regularize  com- 
mercial space  activities 
Need  to  provide  body  of  law  consistent  with  multinational  space  re- 
lated needs 
NASA  Operations  including : 

Contracting  procedures  and  approaches 
Institutional  Support  Contracting 
Facilities  utilization  and  consolidation 
Recruiting,  training  and  retention  practices 
(7-. 
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NASA  Management  including: 

The  "lead  center"  management  concept  for  the  Space  Shuttle 

Planning,  organization,  direction  and  control  of  programs  and 
institutions 

Procurement  management 
NASA  Programs  including: 

Space  Shuttle,  Viking,  LandSat,  ATS 

Tracking  and  Data  Relay  Satellite 

Program  planning  and  control  techniques 

Future  programs — plans,  opportunities  and  alternative  objectives. 

AVIATION 

Agency  duplication  in  Aviation  R&D  programs  and  facilities. 

FAA  and  NASA  R&D  programs :  objectives  and  content. 

Aviation  Safety  R&D. 

Advanced  Aircraft  Systems:  Subsonic,  Supersonic  and  Hypersonic. 

Aviation  Military  Aeronautical  R&D  Relationships. 

Air  Traffic  control :  Long  Term  technology  needs. 

National  Aeronautical  R&D  Facilities  Program. 

Alternative  Aviation  Fuels  and  fuel  conservation  technology. 

Microwave  Landing  System :  Technology  issues  and  status. 

Airport  design  and  development. 


National  Science  Foundation : 

Research  Applied  to  National  Needs :  Program  users  and  utility  of 
programs.  Analysis  of  grant  methodology  of  basic  research 
projects 
Science  Education  programs  and  cutback  effects 
Environmental  data  handling  systems 

R&D  incentive  program  and  its  application 
Development  of  a  National  Science  Policy,  including  the  planning  and 
organizational  structure  for  handling  Federal  R&D  as  well  as  science 
advice  to  the  President 
Strengthening  the  national  economy  through  the  utility  of  Research 

and  Development 
Application  of  Science  and  Technology  to  Urban  Problems 
Materials  Shortages  and  Materials  Research  and  Development 
Technology  Transfer  and  International  Trade 
American-Soviet  Technology  Transfer 
American-Soviet  Science  Cooperation 
Science  in  China 
Federal/local  relations  in  Science  and  Technology 

ENVIRONMENT 

Adequacy  of  inter-agency  coordination  in  environmental  R&D 
Long  term  effects  on  climate  and  health  by  man-made  alterations  of 

physical  and  chemical  characteristics  of  the  atmosphere — including 

ozone  depletion 
Ocean  dumping  and  its  effects  on  marine  biology 
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Lone  term  effects  of  low  level  pollution  of  air  and  water 

Environmental  data  handling  systems 

Staffing  and  direction  of  Federal  laboratories  engaged  in  environ- 
mental research  and  development 

Correlation  of  EPA  research  and  development  activities  with  regula- 
tory functions  prescribed  by  several  statutes 

Evaluation  of  reorganization  of  EPA's  Office  of  Research  and  De- 
velopment 

Development  of  environmental  monitoring  devices — including  those 
for  use  in  aircraft  and  satellites 

Roles  of  industry  and  universities  in  environmental  research  and  de- 
velopment 

ENERGY 

Energy  Research  and  Development  Administration : 

Management  and  the  planning,  organization,  direction  and  con- 
trol of  programs 

Facilities  utilization  and  facilities  litigation 
Energy  R&D  programs  of  other  Agencies — including  EPA,  NSF, 
NASA,  and  the  Departments  of  the  Interior,  Agriculture,  Trans- 
portation, Commerce  and  Defense  (non-military) 
Through  review  of  energy  R&D  laws : 

Public  Law  93^09  The  Solar  Heating  and  Cooling  Demonstra- 
tion Act 

Public  Law  93-410  The  Geothermal  Research,  Development  and 
Demonstration  Act 

Public  Law  93-473  The  Solar  Energy  Research  Development  and 
Demonstration  Act 

Public  Law  93-577  The  Federal  Non-nuclear  Energy  Research 
Development  Act 

The  Coal  Research  Act  of  1960 
Specific  hearings  on  the : 

Relationship  of  price  to  the  development  of  synthetic  fuels 

Methods  for  federal  financing  for  research  and  development 
where  industry  is  involved 

The  role  of  research  and  development  in  developing  a  National 
energy  resource  data  base 

Governmental  planning  for  regional  impacts  of  new  energy  tech- 
nologies 

The  potential  transfer  to  ERDA  of  energy  research  and  develop- 
ment programs  not  transfered  under  the  Energy  Reorganiza- 
tion Act  of  1974 
Research  and  development  of  specific  technologies 

Coal :  mining,  waste  disposal,  combustion,  gasification,  liquefac- 
tion 

Oil  shale,  tar  sands  and  petroleum  and  petroleum  and  natural 
gas  recovery 

Conservation,  transmission,  storage  and  utilization 

Hydrogen,  methonal  and  alternate  fuel 

Federal  power,  solar,  geothermal,  and  advanced  energy  concepts 
Joint  investigations  with  other  subcommittees  and  committees : 

Tax  policies  which  influence  energy  research  and  development 

Land  leasing  aspects  of  geothermal  energy  demonstration  and 
commercial  utilization 
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SPECIAL  OVERSIGHT  REVIEWS 

The  research  and  development  programs  of  the  Federal  Highway 
Administration,  the  National  Highway  Traffic  Safety  Administra- 
tion, the  Federal  Railroad  Administration,  the  Urban  Mass  Trans- 
portation Administration,  and  the  Coast  Guard 
Biomedical  research  and  development  and  aspects  of  medical  and 

genetic  engineering 
Specific  activities  in  civilian  nuclear  energy  research  and  development,: 
Light  water,  high  temperature  gas  and  breeder  reactors 
Controlled  thermonuclear  and  laser  fusion  research 
Nmclear  propulsion  systems 
Nuclear  safety  research 

Environmental  research  related  to  nuclear  energy 
Nuclear  research  and  development  facilities 
This  list  of  specific  and  special  oversight  activities  represents  areas 
of  Committee  interest.  However,  it  does  ,not  commit  the  Committee 
to  examining  each  and  every  one,  nor  does  it  prevent  the  addition  of 
oversight  subjects  determined  by  later  events  and  conditions. 
Sincerely, 

Olin  E.  Teaqtte,  Chairman. 
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House  or  Representatives, 
Committee  ox  Small  Business, 
Washington,  D.C.,  February  18,  i97o. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  UJS.  House  of 
Representatives,  Washington,  D.C. 
Dear  Mr.  Chairman:  This  will  acknowledge  and  respond  to  your 
letter  of  February  13  last  concerning  coordination  of  oversight  activi- 
ties of  all  standing  committees  of  the  House  by  your  Committee. 

There  is  attached  a  listing  by  Subcommittee  of  the  proposed  over- 
sight activities  contemplated  during  the  94th  Congress.  I  have  desig- 
nated Howard  Greenberg,  Executive  Director  of  the  Committee,  as 
our  representative  to  discuss  this  matter  with  your  staff. 
"With  kindest  regards  and  best  wishes,  I  am 
Yery  sincerely  yours, 

Joe  L.  Evens,  Chairman. 

SUBCOMMITTEE   ON   ENERGY  AND  ENVIRONMENT 

This  Subcommittee  will  be  responsible  for  oversight,  investigation 
and  review  of  all  problems  affecting  small  business  arising  from  mat- 
ters concerning  energy  and  the  environment. 

Examples  of  matters  to  be  covered  would  be  marketing  of  petroleum 
products,  transportation  of  petroleum  products,  review  of  natural  gas 
and  synthetic  fuel  development,  coal,  nuclear  energy,  petrochemical 
matters,  and  environmental  problems. 

SUBCOMMITTEE     ON     GOVERNMENT    PROCUREMENT    AND     INTERNATIONAL 
TRADE 

This  Subcommittee  will  be  responsible  for  oversight,  investigation 
and  review  of  all  problems  affecting  small  business  relating  to  gov- 
ernment procurement,  as  well  as  the  problems  of  small  business  in  in- 
ternational trade. 

Examples  of  the  matters  to  be  covered  would  be  procurement  under 
regular  government  procurement  programs.  Small  Business  Act  8(a) 
procurement  by  SBA,  set-asides,  subcontracting,  renegotiation  of  gov- 
ernment contracts,  and  government  regulations  involving  small  busi- 
ness procurement  problems. 

SUBCOMMITTEE   ON    SBA    OVERSIGHT   AND    MINORITY   ENTERPRISE 

This  Subcommittee  will  be  responsible  for  the  investigation  and 
review  of  SBA  operations  and  of  matters  affecting  minority  business 
enterprise. 

(76) 
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SUBCOMMITTEE   ON    ACTIVITIES    OF   REGULATORY    AGENCIES 

This  Subcommittee  will  be  responsible  for  oversight,  investigation 
and  review  of  all  problems  affecting  small  business  relating  to  con- 
centration, monopoly  and  other  matters  involving  regulatory  agencies. 

Examples  of  the  items  to  be  covered  would  be  unfair  and  deceptive 
trade  practices,  advertising  techniques,  credit  regulation,  monopolistic 
practices,  and  antitrust  and  anticompetitive  practices. 

SUBCOMMITTEE   ON    COMMODITIES   AND    SERVICES 

This  Subcommittee  will  be  responsible  for  oversight,  investigation 
and  review  of  all  activities  affecting  small  business  concerning  prod- 
ucts, commodities,  and  services,  except  for  matters  relating  to  energy. 

Examples  are  services  industry  problems,  product  and  commodities 
problems,  grain  marketing  and  transportation,  freight  forwarding 
and  customs  brokerage,  franchising,  red  tape  and  paperwork  burdens 
of  small  business,  and  similar  matters. 
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U.S.  House  of  Representatives, 
Committee  on  Standards  of  Official  Conduct, 

Washington,  D.C.,  February  lJf,  1975. 
Hon.  Jack  Brooks, 

Chairman,  Committee  on  Government  Operations,  House  of  Repre- 
sentatives, Washington,  D.C. 
Dear  Jack  :  This  is  in  reply  to  your  letter  of  February  13,  1975, 
inquiring  about  planned  oversight  activities  by  the  Committee  on 
Standards  of  Official  Conduct,  pursuant  to  Clause  2(c)  of  Rule  X  of 
the  House. 

At  this  time,  our  Committee  plans  to  continue  its  oversight  activities 
in  the  two  areas  in  which  it  has  primary  jurisdiction :  administration 
and  enforcement  of  the  Code  of  Official  Conduct  under  Rule  XLni 
and  Financial  Disclosure  under  Rule  XLIV. 

Our  staff  director,  John  M.  Swanner,  will  be  available  to  confer 
with  your  staff  on  this  matter  at  any  mutually  agreeable  time. 
With  warm  regards,  I  am 
Sincerely, 

John  J.  Flyxt,  Jr.,  Chairman. 
(78) 
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COMMITTEE  ON  VETERANS'  AFFAIRS 


U.S.  House  of  Represk  natives, 
Committee  on  Veterans'  Affairs, 
"Washington,  D.C.,  February  19, 1976. 
Hon.  Jack  Brooks. 

Chairman,  t'o?nniittet   on  Government  Operations,  House  of  Repre- 
s(  ntatives,  Washington,  I>.C. 

Dear  Mr.  Chairman:  This  is  in  response  to  your  letter  of  Feb- 
ruary 13,  1975  calling  attention  to  the  report  that  is  required  "60  days 
after  the  Congress  convenes".  We  are  planning  oversignt  activities  as 
follows: 

1.  Inquiry  into  the  operation  of  the  military  PREP,  tutorial  and 
work-study' programs,  which  are  part  of  the  veterans'  education  and 
training  program.  Particular  attention  will  be  given  to  substantial 
overpayments  which  are  developing  in  the  education  and  training 
program. 

•1.  The  Committee  is  continuously  involved  in  oversight  of  the  hos- 
pital and  medical  program.  We  have  just  completed  a  survey  of  prob- 
lems existing  in  the  system.  We  will  continue  investigating  complains 
relating  to  care,  and  we  are  giving  special  attention  to  the  program 
of  the  Veterans'  Administration  for  the  furnishing  of  certain  prescrip- 
tion drugs.  We  also  monitor  the  Veterans'  Administration  hospital  and 
medical  construction  budget  which  exceeds  $300  million. 

'.).  Comprehensive  oversight  is  planned  in  the  cemetery  field  with  re- 
gard to  adequacy  and  need  for  expansion  and  renovation  of  the  Na- 
tional Cemetery  System. 

4.  Oversight  investigations  are  underway  regarding  the  impact  of 
termination  of  the  '"Vietnam  era"  on  veteran  benefits. 

5.  Several  years  ago  the  guaranteed  loan  program  was  extended  to 
mobile  homes;  oversight  of  the  functioning  of  that  program  is 
planned.  Late  last  year  Congress  extended  the  guaranteed  loan  pro- 
gram to  condominiums,  and  as  that  program  develops,  oversight  ac- 
tivities will  be  necessary.  The  Committee  investigates  complaints 
which  it  receives  regarding  poor  construction  and  bad  sales  or  real 
estate  practices  relating  to  the  veterans'  home  loan  program. 

fi.  Oversight  studios  arc  in  progress  regarding  income  levels  of  needy 
veterans  receiving  pensions.  Studies  are  also  underway  to  make  the 
non-scrvice-connected  pension  program  more  responsive  to  the  acute 
medical  needs  of  the  pensioner. 

Mr.  Oliver  E.  Meadows,  Staff  Director,  will  serve  as  liaison  with 
your  staf:'  representative  and  we  look  forward  to  cooperating  with 
your  Committee. 
Sincerely. 

Ray  Roberts,  Chairman. 
(79) 
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11 1/- '  U.S.  House  of  Representatives, 

;  •'•    .  .  Committee  on  Wats  and  Means, 

Washington,  D.C.,  March  6, 1975. 
Hon,  Jack  Brooks, 
Chairrrvah, 

Committee  on  Government  Operations, 
U.S.  House  of  Representatives 

Bear  Mr.  Chairman  :  This  letter  is  a  follow-up  on  your  letter  and 
the  meeting  between  your  staff  representatives  and  the  Chief  Counsel 
and  Minority  Counsel  of  the  Committee  on  Ways  and  Means  concern- 
ing proposed  oversight  activities  of  the  Committee  on  Ways  and 
Means,  to  comply  with  the  new  rules  of  the  House  for  the  initial  re- 
port on  that  subject  by  your  Committee. 

In  view  of  the  requirements  of  H.  Res.  988,  93rd  Congress,  that 
each  Committee  of  the  House  establish  at  least  four  subcommittees 
and  the  expanded  requirements  of  the  House  rules  as  to  oversight  by 
the  Committees  of  the  House,  the  Committee  on  Ways  and  Means  has 
been  extensively  restructured  in  order  to  meet  those  requirements. 
Thus,  prior  to  the  close  of  the  93rd  Congress,  the  Committee  created 
six  new  subcommittees,  as  follows:  (1)  Subcommittee  on  Social  Secu- 
rity; (2)  Subcommittee  on  Health ;  (3)  Subcommittee  on  Unemploy- 
ment Compensation;  (4)  Subcommittee  on  Trade ;  (5)  Subcommittee 
on  Public  Assistance;  and  (6)  Subcommittee  on  Oversight.  The  full 
Committee  has  reserved  jurisdiction  over  the  Internal  Revenue  Code. 

In  establishing  these  subcommittees,  it  was  the  feeling  of  the  full 
Committee  that  oversight  activities  should  not  be  exclusive  but  should 
be  concurrent,  that  is,  that  the  substantive  subcommittees  should  ex- 
ercise oversight  on  matters  within  their  particular  jurisdiction  and 
that  the  Subcommittee  on  Oversight  likewise  should  have  oversight 
responsibility.  To  this  end,  the  rules  of  the  Committee  provide  that : 
"the  jurisdiction  of  the  Oversight  Subcommittee  shall  include  all 
matters  within  the  scope  of  the  Committee  but  shall  be  limited  to 
existing  law  and  said  oversight  jurisdiction  shall  not  be  exclusive  but 
shall  be  concurrent  with  that  of  the  other  subcommittees  and  with 
respect  to  matters  involving  the  Internal  Revenue  Code  said  con- 
current jurisdiction  shall  be  shared  with  the  full  Committee." 

The  Committee  has  submitted  its  budget  and  we  are  hopeful  that 
in  due  course  it  will  be  approved  by  the  House  so  that  staffing  can 
proceed  for  the  various  subcommittees.  Clearly,  in  view  of  the  re- 
structuring of  the  Committee  and  the  responsibilities  of  the  subcom- 
mittees, if  staffing  proceeds  in  due  course,  the  Committee  on  Ways 
and  Means  will  be  in  a  position  to  exercise  vigorous  and  extensive 
oversight  over  all  the  matters  within  its  jurisdiction. 

(») 
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It  is  contemplated  that  the  Committee  and  its  subcommittees  will 
not  only  provide  continuing  oversight  over  the  general  matters  within 
its  jurisdiction  but  that,  through  its  subcommittees  and  through  the 
Subcommittee  on  Oversight,  it  will  provide  extensive  oversight  into 
specific  areas  of  concentration  in  depth.  Thus,  while  the  Committee 
will  exercise  a  continuing  general  oversight  over  the  Internal  Revenue 
laws,  and  the  Subcommittees  will  carry  out  their  continuing  responsi- 
bilities over  the  Social  Security  Act,  the  customs  and  trade  laws, 
health  programs,  welfare  programs  and  unemployment  compensa- 
tion, we  anticipate  that  the  Subcommittee  on  Oversight  and  each  of 
the  Subcommittees  will  be  selecting  specific  subjects  for  very  careful 
and  deep  analysis  as  time  permits. 

We  anticipate  that  in  social  security  the  following  will  be  involved : 
(1)  administration  and  services  provided  by  social  security  district 
officers;  (2)  social  security  disability  program ;  and  (3)  social  security 
computer  programming;  in  welfare:  (1)  evaluation  of  quality  con- 
trol program  by  HEW;  (2)  administration  of  SSI  program:  and 
(3)  monitoring  implementation  of  new  social  service  legislation:  in 
health:  (1)  the  PSRO  program;  (2)  the  implementation  of  the  renal 
disease  program;  and  (3)  nursing  home  care:  in  trade:  (1)  review  of 
the  collection  and  publication  of  import  statistics;  (2)  preparation 
for  and  conduct  of  negotiations  under  the  Trade  Act  of  1974 ;  and  in 
unemployment  compensation:  (1)  operation  of  the  special  unemplov- 
ment  assistance  program;  (2)  operation  of  the  State  agencies.  The 
Oversight  Subcommittee  has  already  met  with  the  Commissioner  of 
Internal  Revenue  and  has  requested  a  considerable  amount  of  data 
with  regard  to  specific  matters  concerning  the  administration  of  the 
internal  revenue  laws.  We  expect  that  the  oversight  in  this  regard  will 
be  coordinated  with  activities  of  the  Joint  Committee  on  Internal 
Revenue  Taxation  and  with  the  Internal  Revenue  Subcommittee  of 
the  Senate  Finance  Committee. 

In  view  of  our  heavy  schedule,  the  subcommittees  have  not  de- 
veloped complete  agenda  as  to  the  specific  areas  of  concentration, 
although  the  Subcommittee  on  Oversight  has  set  forth  a  number  of 
possible  subjects  of  inquiry,  too  numerous  to  set  forth  here,  to  be  co- 
ordinated with  the  work  of  the  other  subcommittees.  It  should  be 
understood  that  the  Committee  and  subcommittees  will  not  necessarily 
be  limited  to  the  above  listing. 

We  appreciate  this  opportunity  to  advise  you  of  these  activities  of 
our  Committee. 
Sincerely, 

Al  Ullman,  Chairman. 
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OVERVIEW  OE  FEDERAI  AGENCY  PROGRAM  EVALUATION  ACTIVITIES 
JOINT  INQUIRY  BY  THE  OFFICE  OF  MANAGEMENT  AND  BUDGET 
AND  THE  GENERAL  ACCOUNTING  OFFICE 


Introduction 

The  Office  of  Management  and  Budget  (OMB)  and  the  General  Accounting 
Office  (GAO)  share  a  co-nmon  requirement  to  gather  certain  basic  informa- 
tion on  the  activities  and  output  of  program  evaluation  units  within  the 
Federal  government.   This  questionnaire  has  been  developed  jointly  by  OMB 
and  GAO  to  meet  the  information  needs  of  both  groups.   OMB's  main  interest 
is  in  updating  and  expanding  their  current  file  of  descriptive  information 
on  Federal  evaluation  activities,  which  was  begun  in  1974  by  their  Evalu- 
ation and  Program  Implementation  Division,  focusing  on: 

--Identifying  organizational  settings  and  priorities  of  units  for 
which  program  evaluation  represents  a  major  and  continuing 
responsibility; 

--Discerning  the  nature  of  operating  relationships  between  evaluation 
units  and  units  involved  in  related  information  producing  activities 

--Describing  the  processes  employed  to  plan,  coordinate,  and  dissem- 
inate evaluation  studies;  and 

--Obtaining  evaluation  expenditure  and  resource  information. 

The  primary  interest  of  the  GAO  in  this  joint  activity  is  in  respond- 
ing to  requirements  of  the  Congressional  Budget  Act  of  1974.   This  act 
gives  GAO  responsibilities,  shared  with  the  OMB,  the  Congressional  Budget 
Office,  and  the  Secretary  of  the  Treasury  to: 

1.  Develop,  establish,  and  maintain  an  up-to-date  inventory  and 
directory  of  sources  and  information  systems  containing  fiscal, 
budgetary,  and  program-related  data  and  information,  and  a 
brief  description  of  their  content. 

2.  Provide,  upon  request,  assistance  to  committees,  joint  commit- 
tees, and  Members  of  Congress  in  securing  Federal  fiscal,  budg- 
etary, and  program-related  data  and  information  from  the  sources 
identified  in  such  inventory  and  directory. 

The  basic  data  gathered  as  a  result  of  this  inquiry  will  be  organized 
into  a  pilot  file  system  for  evaluative  data  on  agency  programs  by  the 
Congressional  Information  Services  Group,  Financial  and  General  Management 
Studies  Division  of  GAO,  for  use  in  responding  to  congressional  requests. 
OMB  will  utilize  this  same  body  of  information  to  examine  the  way  in  which 
evaluation  efforts  are  structured  and  managed,  and  how  the  results  of  eval- 
uation studies  are  disseminated. 
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Personnel  from  OMB  and  GAO  will  work  closely  in  arranging  for  inter- 
views with  agency  officials  and  will  exchange  the  basic  data  collected  to 
minimize  the  demands  on  the  time  of  agency  evaluation  staffs.   It  is  rec- 
ognized that  some  time  will  be  needed  by  these  staffs  to  collect  certain 
information.   Return  visits  will  be  made  as  needed  for  clarification  of 
questions  or  understanding  of  information  provided. 


Definition  of  Evaluation 

For  the  purpose  of  this  inquiry,  evaluation  is  defined  as  relatively 
structured,  systematic  analysis  of  actual  performance  of  operating  pro- 
grams, and  comparison  of  alternative  programs,  to  assess  the  impact  or 
effectiveness  in  attaining  stated  goals  or  objectives  or  to  assess  their 
efficiency. 

Within  this  broad  definition  are  four  general  categories  of  evalu- 
ation, delineated  generally  on  the  basis  of  their  purpose  and  predominant 
methodology.   These  are;  (1)  substantive  impact  evaluations,  (2)  relative 
effectiveness  evaluations,  (3)  process  or  management  evaluations,  and 
(4)  project  evaluations: 

1.  Substantive  impact  evaluations  attempt  to  measure  the  impact  which 
Federal  programs  have  upon  their  stated  objectives.   This  type  of 
evaluation  seeks  to  determine  what  the  program  accomplishes,  how 
these  accomplishments  compare  to  their  intended  purposes,  and 
their  costs.   The  purpose  of  such  evaluations  is  primarily  to  pro- 
vide information  for  use  in  major  policy  formulation. 

2.  Relative  effectiveness  evaluations  seek  to  compare  the  effective- 
ness of  two  or  more  major  program  strategies  or  approaches  in 
attaining  ultimate  objectives  within  a  national  program.   These 
studies  are  designed  to  help  policy  officials  and  program  managers 
select  the  most  effective  mix  of  services  to  maximize  programs' 
total  impact,  such  as  the  mix  of  skill  training,  remedial  educa- 
tion, and  job  search  assistance  in  a  manpower  program.   However, 
these  studies  do  not  necessarily  measure  the  impact  of  the  total 
program  in  absolute  terms  on  its  objectives. 

3.  Process  of  management  evaluations  are  designed  to  measure  the  oper- 
ating efficiency  of  national  programs.   They  are  intendad  princi- 
pally to  help  program  managers  achieve  the  most  efficient  deployment 
of  available  resources,  rather  than  help  policy  officials  arrive  at 
major  decisions  affecting  the  scope  and  focus  of  the  national 
programs . 

4.  Project  evaluations  may  entail  any  of  the  three  preceding  types 
(substantive  impact,  relative  effectiveness,  or  process  evalua- 
tion) as  well  as  project  rating  --  comparing  the  effectiveness  of 
of  one  or  more  individual  projects  against  others.   Project  eval- 
uations are  directed  to  individual,  locally- based  projects  which 
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are  components  of  a  national  program,  and  are  not  necessarily 
intended  to  yield  conclusions  regarding  the  impact  or  efficiency 
of  the  total  national  program. 


Exclusions 

For  the  purpose  of  this  inquiry,  the  following  information  producing 
activities,  although  related,  are  excluded: 

--Routine  policy  analysis  and  budget  review; 

--Basic  research  not  directed  toward  program  outcomes; 

--Data  collection;  and 

--Audits  for  legal  or  fiscal  compliance  (however,  audits  that  deal 
with  program  results  or  impact  are  of  interest  to  us). 

Definition  of  an  Evaluation  Unit 

Within  each  department  or  agency  being  interviewed,  a  number  of  dis- 
tinct organizational  units  may  be  performing  program  evaluation  as  described 
above.   Each  agency  may  have  a  different  organizational  structure  for  assign- 
ing evaluation  responsibilities.   It  is  desired  to: 

--Identify  and  interview  the  head  of  any  unit  that  has  responsibility 
for  evaluating  the  entire  range  of  agency  programs. 

--Identify  and  interview  the  heads  of  any  units  having  responsibility 
for  evaluating  portions  of  the  entire  range  of  agency  programs. 

This  inquiry  is  directed  to  those  organizational  units  for  which 
evaluation  activities  represent  a  primary  if  not  sole  responsibility. 
This  may  be  a  separate  office  of  a  department  or  agency,  or  a  visible 
organizational  unit. 
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OVERVIEW  OF  FEDERAL  AGENCY  PROGRAM  EVALUATION  ACTIVITIES 
INTERVIEW  GUIDE 

Date: I  i  u-  Number: 

Agency/Department: 

Unit: 

Respondent : Phone  : 

OMB   Staff: Phone: 

GAO  Staff: Phone: 

Person  Completing   Form:   Phone: 

For  questions   or  additional    information  contact- 

OMB  GAO 

Stefan  A.  Halper  Wallace  M.   ohen,  Vssistant  Director 

Rm.  10233  Pm.  500 IB,  Arthur  Bldg. 

Executive  Office  Bldg.  USGAO  -  FCMSD 

Office  of  Management  and  Budget  441  G  St.,N.W. 

Washington,  D.C.  20503  Washington,  D.C.  z05-t6 

Tel.  (202)  395-3612  Tel.  (202)  376-5334 

I.  General  Questions  on  Definitions 

1.  What  is  the  evaluation  unit's  definition  of  program  evaluation? 
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File  Number: 


2.  Are  there  any  substantial  differences  between  this  definition 
and  the  definition  we  set  forth  in  the  introduction? 


3.  Does  this  unit  perform  one  or  ^iore  of  the  four  general  categories 
of  evaluation  set  forth  in  the  introduction?   If  so,  which  ones? 
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II.  Organization  of  Evaluation  Activities 

1.  Name  of  unit : 

2.  Address  of  unit: 

3.  Name  and  title  of  head: 

4.  Telephone: 

5.  Office  to  whom  this  unit  reports: 

6.  Provide  an  organizational  chart  situating  the  evaluation  unit 
being  interviewed  with  respect  to  other  evaluation  units  and 
with  respect  to  agency  structure.   Circle  or  otherwise  designate 
evaluation  units. 
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Provide  the  names  of  other  program  evaluation  units  within  the 
agency  that  are  involved  in  evaluating  similar  programs  as  the 
unit  being  interviewed.  Identify  any  field  components  and  the 
office  to  whom  they  report. 


Where  field  components  have  been  identified,  how  is  control 
exercised?   (e.g.,  setting  of  evaluation  priorities,  evaluation 
scope,  and  quality  control.) 
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9.  Does  the  evaluation  unit  being  interviewed  exercise  any  coordi- 
nating function  with  respect  to  the  activities  of  the  other 
evaluation  units  of  the  agency?   If  so,  how  is  this  done? 


10.  What  are  the  evaluation  unit's  missions  and  areas  of  responsibility' 
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11.  Does  the  unit  perform  evaluation  support  activities?  (e.g.,  main- 
tenance of  evaluation  indexing  or  cross  referencing  systems.) 


12.  Is  there  any  formal  connection  between  this  unit  and  (1)  the 

budget  preparation  process,  or  (2)  the  Management  by  Objectives 
(MBO)  process?   If  so,  describe  the  relationship. 


13.  In  general,  what  proportion  of  this  unit's  evaluation  work  requires 
the  collection  of  new  information  from  the  public?  Circle  best 


estimate. 

0%       25%       50%       75%       100% 
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III.  Delineation  of  Evaluation  Unit's  Program  Responsibilities  and  Functions 

Attached  is  a  listing  of  programs  reported  for  your  agency  in  the 
1974  Catalog  of  Federal  Domestic  Assistance  Programs. 

1.  Does  the  unit  being  interviewed  know  of  any  additional  program 
titles  that  should  be  added  to  this  list  in  order  that  we  have 
a  more  complete  listing  of  programs  administered  by  the  agency? 
For  each  additional  program  not  listed  in  the  Domestic  Catalog, 
provide  the  following: 

A.  Title  of  program 

B.  Authorizing  legislation  (i.e.,  short  title,  U.S.  Statute, 
Public  Law,  and  U.S.  Code) 

C.  Brief  statement  of  program  objectives 

D.  13-digit  budget  account  identification  code(s) 

E.  Any  agency  identification  codes  for  this  program 
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2.  Reviewing  the  listing  of  programs  reported  in  the  Domestic  Catalog 
and  any  additional  program  titles  supplied  for  item  III-l  above, 
indicate  the  programs  for  which  the  unit  being  interviewed  has 
primary  responsibility  for  evaluation 


2/75 


447 


File  Number; 


3.  For  which  programs  listed  in  the  Domestic  Catalog  and  any 

additional  titles  supplied  for  item  III-l  above,  is  evaluation 
authorized? 

Legislation  Authorizing  Eval- 
Domestic  Catalog  Number  or  Program  Title    uation  (Short  Title,  U.S.  Stat.) 
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4.  For  which  programs  reported  in  the  Domestic  Catalog  and  any 
additional  program  titles  supplied  for  item  1II-1  above,  are 
specific  evaluation  studies  or  reports  required? 

Legislation  Requiring  Evaluation 
Domestic  Catalog  Number  or  Program  Title     (Short  Title,  U.S.  Statute  etc.) 
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IV.  Evaluation  Plans  and  Products 

1.  Describe  the  process  by  which  evaluation  activities  are  planned 
and  approved. 
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2.  Attach  copy  of  any  formal  written  plans  describing  evaluation 
activities  for  Fi    75.   If  not  available,  describe  evaluation 
plans  for  FY  75. 
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For  ma  jor  studios  in  process  for  FY  75,  please  list  and  indicate 
estimated  dates  of  completion.  (In  general,  ma jor  is  taken  to  mean 
studies  requiring  more  than  57  of  the  unit's  resources.) 

Estimated  Date 
Tit  le of  Completion 
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4.  What  are  the  major  areas  of  study  planned  for  FY  76?  (i< e.,  studies 
requiring  more  than  57.  of  the  unit's  resources.) 
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5.  Is  there  a  formal  dissemination  plan  for  the  distribution  of 
evaluation  reports,  i.e.,  how  are  recipients  determined? 


6.  Are  any  summary  collections,  digests,  master  lists,  or  other  index 
publications  available  which  indicate:  (1)  evaluation  activity  in 
progress  and/or  (2)  completed  evaluation  reports?   If  possible, 
provide  copies . 
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What  are  the  titles  of  evaluation  reports-  produced  by  the  agency 
in  FY  73,  74,  and  75?   For  work  completed  under  contract,  identify 
the  contractor  and  contract  number.   To  the  extent  that  this  infor- 
mation is  provided  for  item  1V-6  above,  a  repeat  or  reformated 
listing  is  not  required.   The  following  elements  of  information 
should  be  provided: 

A.  Program  evaluation  report  title 

B.  Author (s) 

C.  Publication  date 

D.  Issuing  agency 

E.  Agency  and  retrieval  system  accession  codes  (e.g.,  NTIS 
accession  number) 

F.  Titles  of  programs  that  the  report  evaluates 

G.  Titles  of  programs  to  which  the  report  is  relevant 
H.  By  an  asterisk  or  other  clear  indication,  note  any 

evaluation  which  requires  more  than  57=,  of  the  unit's 
budget  for  the  fiscal  year 


2/75 


455 


File  Number: 


V.  Agency  Evaluation  Budgetary  and  Resource  Information 

The  following  questions  are  directed  towards  determining  the  amount 
of  funds  expended  by  the  evaluation  unit  in  FY  73,  74,  and  75  and  in 
having  an  estimate  of  the  manpower  directed  specifically  towards 
evaluation.   In  the  event  that  the  data  requested  is  not  readily 
available,  a  best  estimate  is  acceptable. 

1.  Agency  or  department: 

2.  Bureau,  office,  administration: 


3.  Evaluation  unit 

4.  Respondent (s): 


Phone: 


Budgetary  Data 


FY  73 


FY  74 


FY  75 


5. Budget  Authorization  ($000) 
for  this  unit 


6.  Obligations  for 
this  unit 


($000) 


A.  Salaries 

($000) 

B.  Contract 

($000) 

services 

C.  Other 

($000) 

7.   Obligations 

($000) 

directed  sol 

ely 

towards  eval 

uation 

activities 

Personnel  Resources 

FY    73 


FY    74 


FY   75 


8.  Total   professional   staff 
for  this   unit 

9.  Professional  man-years 
directed  specifically 
towards   evaluation 

10.   Total  clerical  staff 
for  this   unit 


11.  Clerical  man-years 
directed  specifically 
towards   evaluation 
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Jtebruary  1975 


APPENDIX   C 


GAP  Reports  Which  Include  Evaluation  by  GAP 
or  Which  Deal  With  Agency  Evaluation  Matters 


B-11U37U 
B-UU87U 
B-11837P 

B-11365U 

B-125053 

B-125053 
B-130515 
B-130515 

B-130515 
B-130515 

B-1^30l'+ 

C-133182 


Tate 
Aug.   6,  1973 

Oct.   30,  1973 

March  8,  197U 

March  25,  197U 
Feb.  lU,  197U 

Pet.  5,  1973 

Feb.  20,  1973 

March  1.  1973 

April  2,  19?1* 
May  8,  1973 

July  3,  1973 

Dec.  5,  1973 


Airmail  Improvement  Program 
Objectives  Unrealized 

Observations  on  the  Preferential 
Mail  System 

Improved  Federal  Efforts  Needed  to 
Equally  Consider  Wildlife  Conserva- 
tion With  Other  Features  of  Water 
Resource  Developments 

Telling  America's  Story  to  the  World  — 
problems  and  Issues 

More  Intensive  Reforestation  and 
Timber  Stand  Improvement  Programs 
Could  Help  Meet  Timber  Demand 

More  Usable  Dead  Or  Damaged  Trees 
Should  be  Salvaged  to  Kelp  Meet 
Timber  Demand 

Difficulties  of  the  Neighborhood 
Youth  Corps  In-School  Program  and  its 
Management  Problems 

Activities  of  the  Emergence  Foci  ur;d 
Medical  Services  Project  Alabama 
Council  on  Human  Relations,  Lee  County, 
Alabama 

Restructured  Neighborhood  Youth  Corps 
Out-of-Scheol  Program  in  Urban  Areas 

Evaluation  of  the  Office  of  Economic 
Opportunity's  Performance  Contracting 
Experiment 

Industrial  Management  Review  of  the 
Naval  Air  Rework  Facility,  Alameda, 
California 

Reemployment  Assista.^3  for  Engineers, 
Scientists,  and  Technicians  Unemployed 
Because  of  Aerospace  and  Defense  Cutbacks 
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Number 

Date 

B-133283 

July  30,  1973 

B-IU6285 

April  25,  197U 

B-li+6879 

Jan.   30,  1973 

B-1U6998 
B-151261 

B-159896 

B-159896 

B-16171+0 
B-163058 
B-163375 

B-163375 

B-163375 
" -163375 

B-163375 


Feb.  27,  1973 
Nov.  30,  1973 

April  11,  197U 

Dec.  17,  1973 

Nov.  23,  1973 
.Tune  ?0 .  ]  9"3 

April  k,   197U 

Sept.  2S,  1973 


Jan.  30,  1973 

Jan.  31,  i;-~3 
March  15,  1973 


Title 

U.S.  Foreign  Aid  to  Education: 
Does  Brazil  Need  it? 

Revenue  Sharing:   Its  Use  by  and 
Impact  on  Local  Governments 

Study  of  Federal  Programs  for 
Manpower  Services  for  the  Dis- 
advantaged in  the  District  of 
Columbia 

What  Should  U.S.  Policy  be  for 
Development  Assistance  to  Ecuador? 

Propriety  of  Minimum  Wage  Deter- 
minations for  Clerical  and  Other 
Office  Employees  Under  the  Service 
Contract  Act 

An  Industrial  Management  Review  of 
the  Maintenance  Directorate, 
San  Antonio  Air  Materiel  Area, 
San  Antonio,  Texas 

Industrial  Management  Review  of  the 
Army  Aeronautical  Depot  Maintenance 
Center,  uorpus  Cnristi,  Texas 

Agricultural  Program  Evaluation 
Laws  and  Studies 

Army  Air  i>et>ns<?:   Tn«?  -fAM-D 
Program 

Administration  of  Small  Business  Loan 
Program  Under  the  Occupational  Safety 
and  Health  Act 

Slow  Progress  Likely  in  Development 
of  Standards  for  Toxic  Substances  and 
Harmful  Physical  Agents  Found  in 
Workplaces 

Occupational  Safety  and  Health  Standards 

Occupational  Safety  and  Health  Standards 

More  '^ncerted  Effort  Needed  by  the 
Federal  Government  on  Occupational 
Safety  and  Health  Programs  for  Federal 
Employees 
2  - 
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Report 
Number 

Date 

B-1637U2 

March  21,  1973 

B-16307U 

May  3,  1973 

B-163922 

B-163922 
B-163922 

B-163922 

B-163922 

B-163922 

B-x6403±  (1) 

B-16U031  (2) 
B-16U031  (3) 
B-I6U031  (3) 

B-16U031  (3) 


Sept.  23,  197U 

Aug.  1,  1973 
March  lU,  1973 

June  8,  1973 

Feb.  25,  197U 
March  29,  197^ 

March  7,  197U 

March  11,  197*+ 
April  3,  1973 
June  27,  1973 

June  13,  1973 


Title 

Use  of  Excess  Defense  Articles  and 
Other  Resources  to  Supplement  the 
Military  Assistance  Program 

Advantages  and  Limitations  of 
Computer  Simulation  in  Decision- 
making 

Opportunity  to  Increase  Effectiveness 
of  JOBS -Type  Programs  to  be  Funded 
Under  CETA 

Public  Employment  Programs  in 
Selected  Rural  and  Urban  Areas 

Impact  of  Grants  to  Indian  Tribes 
Under  the  Emergency  Employment  Act 
of  1971 

Public  Service  Benefits  From  Jobs 
Under  the  Emergency  Employment  Act 
of  1971 

Cleveland  Summer  Youth  Employment 
Programs 

The  Emergency  Employment  Act: 
Placing  Participants  in  :;onsubsidized 
Jobs  and  Revising  Hiring  Requirements 

Problems  of  the  Upv/ard  Bound  Program 
in  Preparing  wisadvsnt.agfivi  Sfvudehts 
for  a  Postsecondary  Education 

Progress  and  Problems  in  Providing 
Health  Services  to  Indians 

Effectiveness  of  Vocational  Rehabili- 
tation in  Helping  the  Handicapped 

Social  Services:  Do  They  Help  Welfare 
Recipients  Achieve  Self-Support  or 
Reduced  Dependency? 

Some  Problems  in  Contracting  for 
Federally  Assisted  Child-Care  Services 
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Report 
Number 

B-16U031  (3) 


-16U031(3) 


B-16UU97  (3) 


B-166506 


B-166506 


B-171019 


3-171019 


B-171019 


B-171526 


B-171630 


Date  Title 

June  27,  1973        Problems  in  Administering  Two 

Eligibility  Aspects  --  Incapacity 
and  Unemployment  —  In  the  Aid  to 
Families  With  Dependent  Children 
Program  in  Pennsylvania 

Nov.  20,  1973        Evaluation  of  Efforts  to  Determine 
Nutritional  Health  of  the  U.  S. 
Population 

April  2k,   1973       For  Safer  Motor  Vehicles  —  More 

Effective  Efforts  Needed  to  Insure 
Compliance  With  Federal  Safety 
Standards 

Nov.  15,  1973        Improved  Federal  and  State  Programs 
Needed  to  Insure  the  Purity  and 
Safety  of  Drinking  Water  in  the 
•  United  States 

Jan.  l6,  197^        Research  and  Demonstration  Programs 
to  Achieve  Water  Quality  Goals: 
What  the  Federal  Government  Needs 
to  do 

Oct.  21,  197*+  Progress  in  Determining  Approaches 
Which  Work  in  the  Criminal  Justice 
System 

Jan.  l63  1973       Development,  of  a  Nationwide  Criiiunai 

Data  Exchange  System  —  Need  to 
Determine  Cost  and  Improve  Reporting 

March  19 ,  I97U       Difficulties  of  Assessing  Results 
of  Law  Enforcement  Assistance 
Administration  Projects  to  Reduce  Crime 

May  22,  1973         Interim  Report  on  the  Agency  for 

International  Development's  Housing 
Investment  Guaranty  Program 

March  28,  197*4-       Observations  on  Housing  Allowances 

and  the  Experimental  Housing  Allowance 
Program 
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Report 
Number 

Date 

B-171630 

March  27,  197U 

B-172707 

Aug.  13,  1973 

B-175155 

Aug.  22,  1973 

B-17601+9 

Jan.  22,  1973 

B-177011 

Aug.  lU,  1973 

B-177166 

June  12,  1973 

B-177952 

May  2,  1973 

9-178561* 

June  29,  1973 

B-1737U1 

Nov.  29,  197U 

(KED-75-31C) 

Dec.  18,  197- 

Title 

Problems  in  the  Homeowner  ship 
Opportunities  Program  for  Low- 
Income  Families 

The  .Army  Reorganization  for  the  1970s: 
An  Assessment  of  the  Planning 

Railroad  Reservation,  Information 
and  Ticketing  Services  3eing  Improved 

Need  to  Improve  Language  Training 
Programs  and  Assignments  for  U.S. 
Government  Personnel  Overseas 

Federal  and  State  Efforts  to  Control 
Water  Pollution  Caused  by  Acid  Drain- 
age From  Mines 

In-FIight  Escape  Systems  for  Heli- 
copters Should  be  Developed  to  Prevent 
Fatalities 

Problems  in  Meeting  Military  Manpower 
Needs  in  the  All-Volunteer  Force 

Progress  and  Problems  in  Achieving 
Objectives  of  School  Lunch  Program 

Employment  Services  for  Vietnam  — 
Era  Veterans  Couxd  be  Improved 

Special  Supplemental  Food  Program 
Food  and  Nutrition  Service 
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■  LiLlll-JJzlL  ' ' ''iwal    Act ■oiiiit  i  1 1^  Ol  f  ic<-  Reports 
Involv  in^  I'ro^rain  Evaluation 

In  its  reviews  of  a  variety  of  Federal  programs,  t  lie  General 
'counting  Office  has  conducted  evaluations  and  lias  reviewed  the  validity 
of  agency  evaluations.   Although  problems  have  been  found  and  reported, 
the  overall  tone  has  been  positive  regarding  the  potential  value  of 
program  evaluation,  as  the  following  selected  examples  show:   (Underscoring 
to  emphasize  positive  points.) 

1.  In  Observations  on  the  Preferential  Mall  System,  B-114874, 
October  30,  1973,  GAO  made  use  of  a  computer  model  to  calculate 
alternative  costs.  The  report  explains  the  reasons  for  our  corrections 
to  what  we  believe  to  be  "questionable"  assumptions. 

2.  In  Social  Services;   Do  They  Help  Welfare  Recipients  Achieve 
Self-Support  or  Reduced  Dependency?,  B-164031(3),  June  27,  1973,  we 
stated  that  we  used  an  approach  which  could  give  case  workers  a  more 
systematic  and  analytic  means  to  assess  recipients  potential  and, 
accordingly,  which  could  improve  the  allocation  of  social  service 
resources.   We  stated  that  this  approach  was  developed  by  the  Denver 
Department  of  Welfare  and  our  report  contains  a  chapter  dscussing  this 
method  and  concluding  that  it  is  feasible  to  develop  a  predictive 
approach  using  recipient  characteristics  and  that  further  refinement 
of  the  approach  could  be  the  next  step. 

3.  In  Evaluation  of  the  Office  of  Economic  Opportunity's 
Performance  Contracting  Experiment,  B-130515,  May  8,  1974,  GAO 
demonstrates  the  capability  to  assess  a  social  experiment  which  included 
a  built-in  evaluation  design.   This  can  viewed  as  positive  with  regard 
to  evaluation  because  the  problems  reported  are  in  the  implementation 
more  than  in  the  design.   The  digest  states  that  observations  contained 
in  this  report  are  expected  to  be  of  value  to  the  Institute  (National 
Institute  of  Education)  and  local  educational  authorities  if  similar 
experiments  are  conducted  in  the  future. 

4.  In  Maternal  and  Child  Health  Programs  Authorized  by  Title  V, 
Social  Security  Act,  B-164031(3),  June  23,  1972,  on  program  evaluation 
systems  for  these  projects  which  points  out  some  problems  in  implementa- 
tion but  ends  on  a  positive  note  "on  the  basis  of  our  review  of  the 
system's  general  design,  we  believe  that  it  is  capable  of  producing 
information  useful  to  program  managers  in  measuring  project  and  program 
effectiveness."  The  report  contains  specific  recommendations  for  actions 
which  might  improve  the  system. 
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5.  In  Improvements  Needed  in  the  Administration  of  Contracts 

for  Evaluations  and  Studies  of  Antipoverty  Programs,  B-130515,  December  28, 
1971,  problems  are  stated  and  specific  recommendations  are  given  for 
Improving  both  contracts  and  monitoring,  to  develop  and  implement  a 
system  to  Insure  the  formal  assessment  of  the  results  of  the  evaluations 
and  studies,  and  the  effective  utilization  of  the  results  of  contractors' 
studies  to  Improve  the  design  and  administration  of  programs  or  to 
develop  new  program  approaches  when  appropriate. 

6.  In  Need  for  Improving  the  Administration  of  Study  and 
Evaluation  Contracts,  B-164031(l),  August  16,  1971,  containing  problems 
and  recommendations  with  regard  to  education  program  evaluations  similar 
to  the  report  on  antipoverty  programs  (see  above  item). 

7.  In  Observations  on  Housing  Allowances  and  the  Experimental 
Housing  Allowance  Program,  B-171630,  March  28,  1974,  GAO  has  highlighted 
some  of  the  issues  involved  in  evaluating  such  a  program  including 
studies  conducting  by  a  GAO  consultant.   GAO  recommends  that  the  Congress 
in  considering  future  legislation  authorizing  a  national  housing  allowance 
program,  weigh  the  benefit  that  could  be  derived  from  waiting  until  the 
experimental  program  is  complete  and  more  information  is  available  on 

the  likely  Impact  of  such  a  program.   Thus,  this  report  is  a  positive 
point  of  review  regarding  the  benefit  of  evaluation  and  of  waiting  for 
such   evaluation  results  for  use  in  major  decisions.   This  report  is 
similar  to  the  staff  study  and  testimony  provided  to  the  Senate  Pinance 
Committee  in  1970  on  evaluation  of  the  New  Jersey  negative  income  tax 
experiment. 

8.  In  Progress  in  Determining  Approaches  which  Work  in  the 
Criminal  Justice  System,  B-l 71019,  October  21,  1974,  GAO  recommends 
that  the  Law  Enforcement  Assistance  Agency  should  issue  guidelines 
requiring  state  plans  which  discuss  how  evaluation  will  be  used  to 
assist  in  management  decisions  and  the  extent  to  which  administrators 
believe  current  evaluation  strategies  need  to  be  modified  to  make  this 
possible.   The  report  also  states  that  there  is  no  doubt  that  outcome 
evaluation  is  complicated  and  in  some  instances  costly.   The 
consequence  of  not  doing  such  evaluations,  however,  is  to  reduce  the 
planning  process  to  chance.   Evaluations  are  necessary  so  more 
objective  decisions  can  be  made  regarding  allocations  of  resources. 

9.  In  Difficulties  of  Assessing  Results  of  Law  Enforcement 
Assistance  Administration  Projects  to  Reduce  Crime,  B-171019, 

March  19,  1974,  GAO  recommends  designation  of  demonstration  projects 
to  determine  information  that  should  be  gathered  and  the  type  of 
evaluation  that  should  be  done  and  the  standardized  methodologies  and 
standard  data  reporting  systems  to  accomplish  this. 
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APPENDIX  D 


Program  Evaluation:  A  Manual  for  Legislators 
and  Legislative  Staffs 


Draft  of  January  1975 


Prepared  by  William  H.  Foskett,  Consultant,  and  Harrison  W. 
Staff  of  Senator  Bill  Brock,  (Tennessee) 


Comments  on  the  draft  are  invited  and  should  be  sent  to 
Harrison  W.  Fox,  Jr.  Office  of  Sen.  Bill  Brock, 
254  Russell  Senate  Office  Building,  Washington,  D.C.   20510 
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INTRODUCTION 

Finding  and  usinr  the  links  between  legislative  r.onls,  govern- 
ment spending,  program  activity,  and  consequences  are  at  the  heart 
of  Congressional  authority.   Failure  to  make  strong,  valid,  links 
Invites  the  consequences  of  an  uncontrolled  budget  and/or  unsolved 
human  and  social  problems.  The  Legislative  Reorganization  Act  of 
1970  and  the  Congressional  Budget  and  Impoundment  Control  Act  of 
197*4  in  principle  provide  Congress  with  an  information  resource 
mechanism  and  a  budget  procedure  necessary  "to  exert  its  power  of 
the  purse  with  full  knowledge  of  the  consequences  of  its  actions". 

IT  Congress  wants  to  exert  its  full  authority  under  these 
Acts  it  will  have  to  consider  the  human  and  social  consequences, 
as  well  as  the  economic  consequences,  of  its  actions.   Practically 
speaking,  this  will  require  Congress  to  obtain  and  use  evidence 
which  links  goals,  spending,  program  activities  and  program  results 
to  each  other.   In  setting  national  priorities  and  working  within 
self-imposed  budget  limits  Congress  will  need  solid  evidence 
about  the  cause-effect  relationships  between  program  results  and 
Congressional  goals.   Cause-effect  relationships  can  be  verified 
only  within  the  logic  of  experimentation,  (e.g.  the  negative 
income  tax  experiment),  thus  Congress  will  need  to  provide  for  such 
experiments  as  it  seeks  answers  to  questions  about  what  services 
will  cause  the  results  it  seeks. 

Causality  is  not  the  substance  of  every  legislative  question. 
So  Congress  will  also  need  to  use  other  non-experimental  evaluation 
methods — e.g.,  needs  assessment. 
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We  aim  to  provide  legislators  and  their  staffs  with  sufficient; 
knowledge  of  evaluation  research  to  maintain  their  political 
autonomy  while  focusing  on  the  best  possible  evidence Jfcr  the  legi- 
slative questions  at  hand.   This  manual  does  not  aim  to  make  the 
reader  into  an  instant  expert  on  highly  technical  matters  of  evalu- 
ation.  Evaluation  users  should  keep  in  mind  that  serious  limitations 
currently  exist  in  the  organizational  resources  available  to  supply 
such  evidence  and  in  the  state  of  the  art  of  evaluation  research. 

By  taking  the  steps  listed  below  you  can  build  a  sound  evidence 

base  for  deciding  how  to  vote  on  authorizations  and  appropriations. 

In  addition,  you  can  build  an  evidence  base  for  strong  committee 

oversight  of  legislation.   Each  of  these  steps  will  be  described 

in  the  text  of  this  manual. 

Step  1:   Selecting  questions  to  be  answered  and  evaluation 
techniques  to  be  used, 


Step  2 
Step  3 
Step  H 
Step  5 
Step  6 
Step  7 


Drafting  legislative  language. 
Formulating  program  goals  and  terms 
Measures. 
Design. 

Statistical  analysis. 
Format  for  evaluation  studies. 


Step  1:   Selecting  questions  to  be  answered  and  evaluation  tech- 
niques to  be  used.   There  are  some  simple  general  types  of  questions 
which  have  profound  practical  meaning  for  effective  legislation. 
Particular  instances  of  these  general  questions  being  asked  are 
common  to  most  legislation  at  some  phase  of  its  life.   Such  questions 
are  listed  across  the  top  of  Table  1.   There  is  at  least  one  or 
more  evaluation  technique (s)  which  can  supply  evidence  in  response 
to  each  of  these  questions.   A  basic  list  of  the  categories  of 
these  techniques  is  shown  down  the  left  side  of  the  table. 
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Table  1 
(Being  Developed) 


LEGISLATIVE 
QUESTIONS 


Your  Suggestions  are  welcome 
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Many  questions  will  come  up  during  the  life  of  an  Act  — 
from  its  introduction  as  a  bill,  to  its  consideration  for  re- 
authorization.  You  should  act  to  see  that  the  type  of  evaluation 
research  that  matches  these  questions  is  beg*n  early.   It  is 
important  to  note  that  the  selection  of  an  evaluation  strategy 
must  in  most  cases  be  completed  before  program  implementation  begins. 

Today,  most  legislatively-mandated  and  executive-initiated 
evaluation  is  not  framed  so  as  to  provide  answers  to  basic  legi- 
slative questions  which  arise  especially  in  oversight.   Indeed 
these  provisions  seem  to  satisfy  mainly  executive  branch  interests. 
The  most  common  Congressional  control  device  is  to  be  found  in 
the  lanfruap-o  which  deals  with  measures  of  effectiveness  and/or 
the  nature  of  recommendations  to  be  made  to  the  Congress.   (Examples 
of  such  legislative  language  may  be  found  in  Program  Evaluation: 
Legislative  Language  and  a  Users  Guide  to  Selective  Sources, 
Part  2.   U.S.  General  Accounting  Office,  1973.) 

Though  there  are  many  laudible  features  in  this  language,  such 
provisions  for  evaluation  have  typically  failed  to  provide  Congress 
with  the  kind  of  information  it  needed  at  the  time  it  needed  it. 
Often  such  language  has  been  self-defeating  by  blurring  the  purpose 
of  evaluation,  or  by  failing  to  authorize  the  necessary  conditions 
(e.g.,  randomization")  or  resources  to  obtain  the  evidence. 

In  the  next  section  you  w^.11  find  legislative  language  options 
along  a  number  of  crucial  dimensions  which  can  determine  evaluation 
activity.   By  selecting  appropriate  options  the  drafter  of  legi- 
slation can  clearly  authorize  and  provide  an  opportunity  for  the 
collection  of  evidence  necessary  to  answer  the  right  questions  at 


the  right  time,  with  the  right  evaluation  technique. 

Step  2:   Drafting  legislative  language.   Congress  often  finds 
that  it  doesn't  have  the  right  kind  of  information  at  the  right 
time,  even  though  an  extensive  "report"  may  have  been  completed. 
The  simplest  and  most  common  reason  for  this  is  that  the  legi- 
slative language  of  an  act's  reporting  sections  provides  little 
or  no  guidance  for  Congressional  evaluation  needs.   The  language 
of  the  reporting  section  provides  a  basic  foundation  for  oversight. 
Effective  Congressional  oversight  is  to  a  large  extent  dependent 
upon  the  specification  of  a  reporting  section. 

If  the  language  in  the  reporting  section  is  too  specific, 
Congressmen  or  their  staff  may  be  entangled  with  technicalities 
or  petty  details  of  planning  or  managing  evaluation  research. 
If  the  language  is  too  vague,  other  Interests  may  pre-empt  Congress 
need  for  timely,  pertinent,  and  credible  oversight  information. 

There  are  eight  aspects  of  evaluation  which  Congress  should 
deal  with  in  the  language  of  reporting  sections  of  legislation: 

-  purpose(s)  of  evaluation 

-  who  will  conduct  the  evaluation  research 

-  content  of  evaluation 

-  the  timing  of  evaluation 

-  funding  for  evaluation 

-  congressional  veto 

-  to  whom  evaluation  information  will  be  reported 

-  access  to  information 
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A  variety  of  legislative  choices  are  available  in  each  of 
these  aspects  of  evaluation.   Legislative  language  for  the  eight 
options  is  shown  in  Table  2. 
Step  3:   Formulating  program  goals  and  terms. 

In  this  step  (Step  3)  and  in  Steps  4,5,  and  6,  we  will  deal 
with  matters  which  aim  to  give  you,  the  evaluation  user,  sufficient 
control  over  the  specifics  of  evaluation  planning  to  see  that  the 
type  of  evidence  you  need  will  be  supplied  by  the  evaluation. 

A  description  of  the  program  in  clear  concrete  terms  is  an 
essential  condition  for  evaluation.   The  program  goals  and  services 
must  be  described  in  such  a  way  that  they  can  be  validly  measured — 
even  if  by  relatively  primitive  techniques.   Someone,  either  the 
drafters  of  the  legislation  or  the  program  planners,  must  produce 
a  concrete  description  of  the  program  which  translates  Congressional 
goals  into  measurable  objectives. 

For  example,  "improving  job  opportunities  for  minorities" 
will  not  be  a  measurable  goal,  whereas  "reducing  the  unemployment 
rate"  or  "increasing  the  percentage  of  minority  group  members  in 
management,  professional,  and  skilled  labor  positions"  would  be 
measurable  goals. 

Congress  has  been  frequently  criticized  by  professional 
evaluators  for  not  writing  the  goals  of  legislation  in  measurable 
terms.   You  should  not  leave  the  definition  of  Congressional  intent 
to  those  responsible  for  the  technicalities  of  evaluation  research. 

It  will  often  be  impractical  (for  passage  and  implementation) 
to  burden  a  bill  with  the  level  of  technical  detail  most  convenient 
for  the  evaluator.   Such  detailing  may  be  a  barrier  to  passage  of 
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n  hill;  or  If  passed  may  lead  to  a  program  lacking  sufficient 
flexibility  for  successful  management.   We  believe  that  you  can 
solve  this  problem  by  systematically  defining  in  the  legislative 
history,  preferably  in  the  committee  or  conference  report,  the 
elements  of  the  program  or  by  requiring  that  a  formal  program  plan 
be  prepared  by  the  program  planners  and  submitted  for  legislative 
review  before  implementation  of  the  program  can  begin — i.e.,  the 
Congressional  veto  concept  applied  to  the  program  plan. 

Full  measurable  descriptions  of  objectives  and  actions  should 
be  required  in  program  plans  if  you  intend  to  consider  implementation 
or  effectiveness  of  the  program, 

The  actual  technical  manpower  necessary  to  develop  these 
elements  of  program  description  in  legislative  or  formal  program 
plans  may  be  obtained  from  outside  contractors,  Congressional 
Budget  Office,  Congressional  Research  Service,  Office  of  Program 
Review  and  Evaluation,  the  General  Accounting  Office,  or  Executive 
Branch  agencies  as  authorized  by  the  Congressional  Budget  and 
Impoundment  Control  Act  of  1971*. 

In  the  committee  or  conference  report  you  should  also  state 
feared,  undesirable  side-effects  which  evaluators  should  look  for 
and  measure — in  addition  to  the  desirable  results  expressed  in 
program  goals.   Opponents  of  a  bill  will  often  be  very  astute 
sources  of  undesirable  side-effects. 

Step  k :      Measures . 

After  formulating  program  goals  and  defining  major  terms, 
one  must  develop  the  valid  measures  for  use  in  the  program  evaluation. 
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These  measures  will  supply  the  basic  body  of  evidence  for  drawing 
conclusions  about  program  effectiveness.   For  Instance,  n  program 
to  Improve  highway  safety  might  have  as  Its  p;oal  to  reduc-p  the 
accident  rate,  to  reduce  personal  injury  as  well  as  the  severity  of 
personal  injury,  and  death,  to  reduce  the  extent  and  severity  of 
property  damage  or  any  combination  of  these.   A  question  which 
still  must  be  answered  is:   How  are  these  goals  to  be  measured? 
Such  a  program  might  have  the  anticipated  (thus  measured)  undesirable 
cost  of  increasing  commuting  time,  unemployment  due  to  inability 
to  commute  to  work,  etc. 

It  is  very  important  to  choose  measures  in  advance  to  be  sure 
that  they  will  supply  the  necessary  data  on  which  oversight  should 
be  based.   Legislative  users  of  evaluation  should  ask  the  following 
questions  about  measures  when  reviewing  plans  setting  up  programs 
and  evaluating  their  effectiveness. 

(1)  Are  the  basic  ideas  of  the  program  adequately  translated 
into  measurable  terms?  (Are  you  really  satisfied  that  the  goals 
and  objectives  of  highway  safety  are  adequately  measured?) 

(2)  Does  the  measurement  cover  program  input?  Are  they 
adequate  to  verify  that  the  program  has  in  fact  been  implemented? 
Do  they  provide  quantitative  measures  of  this  input? 

(3)  Will  the  data  be  sufficient  to  answer  all  of  the  important 
questions  which  you  should  raise  in  oversight  of  the  program's 
effectiveness?  fDo  not  be  caught  short  without  measures  to  answer 
Important  highway  safety  questions [ ) 

CO  Is  each  measuring:  instrument  sensitive  enough  to  show  the 
magnitude  of  differences  which  the  program  is  expected  to  create 
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both  services  and  effects?   (If  you  are  expecting  to  improve 
highway  safety  by  inches,  don't  try  to  measure  that  improvement 
by  feet . ) 

(5)  On  the  whole,  do  the  measurements  proposed  for  each  aspect 
of  program  treatments  and  program  effects  reflect  the  idea  which 
Congress  intended  the  program  to  translate  into  reality?   (If 

the  goal  is  to  reduce  highway  accidents  during  rush  hours,  collect 
the  data  at  appropriate  time  intervals — daily,  not    weekly, 
monthly  or  yearly.) 

(6)  Do  the  items  composing  particular  measurements  of  parti- 
cular program  treatments  or  effects  fairly  sample  all  of  the  im- 
portant aspects  of  that  particular  treatment  or  effect?  Make  sure 
that  all  aspects  of  highway  safety  that  are  perceived  to  be  im- 
portant are  being  measured.   Are  the  undesirable  side  effects 
represented  in  the  measures?   (For  instance,  highway  safety 
affects  property,  lives,  insurance  rates,  birth  rates,  and  food 
consumption.   But  probably  the  first  three  are  of  interest.) 

There  are  rarely  simple,  definitive  answers  to  the  above 
questions.   But  by  raising  these  questions  and  reviewing  their 
answers,  the  legislative  user  of  evaluation  can  gain 

evidence  necessary  to  estimate  the  soundness  of  proposed  measures. 
Developing  measures  is  only  one  step  in  the  evaluation  process. 
The  next  step  the  legislative  evaluation  user  should  take  is  to 
examine  the  design^ plan  for  collecting  data  on  program  services 
and  effects. 
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Step  5-      Design 

The  evaluation  design  is  a  plan  for  linking  measures  of 
program  services  and  results. 

Good  measures  and 
sound  evaluation  design  evaluation  are  both  necessary  for  effective 
program  evaluation  of  programs.   A  well  conceived  evaluation  design 
will  allow  effectiveness  to  be  attributed  to  the  program  itself 
rather  than  to  events  or  circumstances  outside  the  program. 

Weakness  in  design  and  measurement  tend  to  obscure  positive 
evidence  of  program  effectiveness. 

It  is 
very  much  in  the  interest  of  proponents  of  particular  programs  to 
obtain  the  most  valid  measurement  instruments  and  most  powerful 
research  design  available  in  order  to  prevent  extraneous  factors 
from  obscuring  program  effectiveness. 

While  it  will  not  usually  be  appropriate  for  Congress  to 
specify  in  legislation  the  details  of  research  design  and  statis- 
tical analysis,  there  are  a  number  of  important  design  features 
which  are  under  legislative  control  but  which  in  past  legislation 
have  often  been  omitted  or  inadvertently  precluded. 

Evaluation  research  design  is  extremely  complex.   It  requires 
many  technical  judgements  which  cannot  be  made  responsibly  by  a 
layman  untrained  in  research  design.   But  legislators  cannot  meet 
their  responsibility  by  merely  surrendering  the  whole  matter  of 
validity  to  experts.   Legislators  have  a  responsibility  to  raise  basic 
questions  about  the  validity  of  the  evidence  provided  by  the  program 
evaluation  they  authorize..   You  can  meet  this  responsibility  by 
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requiring:  that  evaluation  experts  supply  straightforward  answers 

to  the  basic  threats  to  valid  evaluation  found  at  the  design  stage.* 

Table   3  below  states  these  threats  in  question  form: 

Table  3 
QUESTIONS 

Issues  In  Evaluation  Design  Validity 

1.  Will  the  evaluation  supply  evidence  that  the  program  services 
were  actually  delivered  in  the  intended  type,  quantity  and  quality? 

2.  During  evaluation  of  the  program  can  events  outside  the  program 
blunt  or  exaggerate  measures  of  program  effects?  (history) 

3.  Could  natural  changes  or  pre-existing  trends  (e.g.,  normal 
growth  or  losses)  that  would  occur  regardless  of  the  program  account 
for  results?  (maturation) 

4.  Can  the  process  of  measuring  itself  induce  change  in  that  which 
Is  being  measured?  (testing) 

5.  Can  the  measures  themselves  change  in  meaning  during  evaluation 
of  the  program?  (instrumentation) 

6.  If  comparison  groups  are  used,  would  they  have  changed  differ- 
ently from  each  other  even  if  they  had  not  received  different 
program  services?  (selection) 

7-  Are  participants  selected  on  the  basis  of  extreme  scores  on 
measures  related  to  program  effectiveness  measures?  (statistical 

regression) 

8.  Will  the  evaluation  show  whether  participants  who  drop  out  of  the 
program  (or  different  comparison  groups)  are  alike  in  some  way 
which  would  influence  program  effectiveness  measures?  (mortality) 

9.  To  what  extent  can  valid  evaluation  results  (positive  or  negative) 
be  generalized  to  other  possible  participants  or  program  settings? 
(interaction  of  testing  reactive  arrangements) 

10.  Is  variation  in  program  services  within  critical  range  speci- 
fied for  the  program?  (see  §1   above) 

11.  Are  several  different  ways  of  measuring  each  program  effect 
used  so  that  corroborative  evidence  of  results  Is  provided? 
(generalizing) 

*More  detailed  examination  of  various  design  consideration  can  be 
found  in  Appendix  C. 
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Between  Steps  5  (Evaluation  Design^  and  6  (Statistical  Analysis) 
the  program  evaluation  strategy  will  be  implemented.   Committee 
members  and  staff  should  keep  a  close  watch  over  these  tactical 
activities  of  program  evaluation.   Such  techniques  as  formal 
reporting  requirements,  informal  contact,  etc.,  should  be  used  to 
keep  tabs  on  data  collection  procedures,  timing  of  evaluation 
activities,  program  content,  and  decision  making  needs. 

Step  6.   Statistical  Analysis 

Previous  steps  we  have  recommended  all  aim  to  provide  the 
soundest  possible  raw  data  base  for  oversight  of  program  effective- 
ness.  These  steps  minimize  the  possibility  that  the  basic  evidence 
regarding  program  effectiveness  is  confounded  by  inadequate  program 
delivery,   poor  measurement,  or  invalid  research ^design.   Next  this 
valid  data  must  be  condensed  and  focused  on  the  issues  to  be  raised 
in  oversight  by  use  of  statistical  analysis.   Given  valid  measure- 
ments made  in  a  design  framework  which  yields  interpretable  data 
there  may  be  more  than  one  right  way  to  statistically  analyze  the 
data. 

Statistical  analysis  may  be  used  to  describe  raw  data  or  to 
draw  Inferences  about  relationships  and  meaning  of  the  data.   Used 
descriptively  statistical  analyses  condense  large  amounts  of  raw 
data  In  such  a  way  that  basic  characteristics  are  pointed  out  and 
described  in  succinct,  numerical  terms.   Used  inferentially 
statistical  analysis  searches  out  and  tests  relationships  between 
different  measures — especially  between  measures  of  program  activities 
and  program  effectiveness.   The  variety  of  descriptive  and  in- 
ferential statistical  techniques  used  in  program  evaluation  is  so 
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varied  and  broad  that  judgement  as  to  the  proper  use  of  any  par- 
ticular statistical  technique  ±n   any  specific  situation  must  be 
left  to  the  Professional  evaluator.   But  legislative  users  of 
effectiveness  evaluation  should  take  a  number  of  precautions  to 
minimize  entry  of  bias  into  results  at  the  point  of  statistical 
analysis — or,  in  cases  where  bias  is  unavoidable,  to  make  that 
bias  apparent. 

The  strategy  we  recommend  to  legislative  users  for  dealing 
with  statistical  analysis  has  two  elements: 

-  documentation  of  four  types  of  information  (cited  below) 
about  the  statistical  analyses  used  in  effectiveness 
evaluation  reports; 

-  Independent,  simultaneous  statistical  analyses  of  the  data 
using  alternative  techniques. 

By  documenting  the  following;  four  basic  considerations  for 
statistical  analysis  in  effectiveness  evaluations  professional  and 
public  attention  can  be  conveniently  drawn  to  areas  where  problems 
have  been  overlooked  in  the  past.   Such  documentation  should  be 
clearly  labeled  and  appended  to  the  effectiveness  evaluation  report. 
Of  course,  each  such  appendix  would  implicitly  require  evaluators  to 
take  and  report  certain  precautions  that  might  otherwise  be  over- 
looked or  unreported.   Each  of  the  topics  below  should  be  documented 
in  an  appendix. 

a.  Incomplete  or  missing  data  should  be  screened  and  treated 
in  such  a  way  as  to  avoid  biasing  the  results.   Because  such 
procedures  are  basic,  no  chance  should  be  taken  that  they  might 
be  skipped  over  in  haste  or  economy. 

b.  All  statistical  techniques  used  to  work  with  the  data  should 
be  documented.   Each  specific  technique,  its  assumptions,  evidence 
that  these  assumptions  are  met  in  each  case  of  its  use,  and  page 
references  to  use  of  the  technique  in  the  evaluation  report  (in- 
cluding summaries)  should  be  cited. 
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c.  The  limits  of  findings  based  on  statistical  inference 
inherent  in  the  particular  data  and  the  particular  statistical 
techniques  supporting  them  should  be  clearly  and  concisely  stated. 
Confidence  intervals  and  explicit  description  of  the  time,  setting 
and  participants  to  which  the  findings  can  be  applied  should  be 
included . 

d.  Ways  in  which  the  size  or  direction  of  program  effective- 
ness evaluation  results  might  be  an  artifact  of  statistical 
technique—for  example,  "statistical  significance"  —  should  be 
briefly  indicated  to  give  the  legislative  user  an  indication  of 
possible  practical  weakness  in  the  evaluation  findings.   Ideally, 
statistical  analysis  itself  would  not  affect  apparent  results — 
but  given  complex  situations,  relatively  small  effects,  and  more 
than  one  right  choice  for  the  statistician  to  make,  it  is  worth- 
while either  explicitly  to  rule  out  such  problems  or  to  indicate 
their  extent  in  an  appendix. 

The  second  element  of  the  strategy — independent,  simultaneous 

statistical  analyses  of  the  data  using  alternative  techniques — 

can  keep  the  legislative  user  from  becoming  the  captive  of  a  frame 

of  reference  and  assumptions  which  he  may  not  accept.   If  the 

findings  of  alternative  analyses  of  the  data  tend  to  coincide, 

corroborative  evidence  is  provided;  or,  if  they  diverge  it  is 

perhaps  the  issues  surrounding  program  effectiveness  which  should 

be  debated  and  not  the  degree  of  effectiveness.   Of  course,  all 

alternative  statistical  analyses  would  have  to  be  responsibly 

conducted  and  documented  as  recommended  above.   Independent, 

simultaneous  statistical  analyses  are  relatively  inexpensive  compared 

to  the  cost  of  developing  measurement  instruments  and  collecting 

the  raw  data.   In  practical  terms  it  would  require  that  those  who 

actually  collect  the  raw  data  store  it  on  computer  tapes  in  such 

a  way  that  they  are  readily  usable  by  other  professional  evaluators 

to  whom  the  raw  data  would  be  made  available  for  alternative  analyses 

In  the  past,  alternative  analyses  of  raw  data  have  been  made 
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sequentially  rather  than  simultaneously  with  the  result  that  their 
finding  have  not  been  available  for  corroboration  of  results  or 
sharpening  of  Issues  when  needed  by  legislative  users.   And,  such 
sequential  analyses  of  data  gathered  at  one  point  in  time  may  take 
so  long  that  the  raw  data  are  obsolete  with  respect  to  present  or 
future  situations  with  which  the  legislative  user  may  be  concerned. 
Any  alternative  statistical  analyses  should  be  made  simultaneously . 

Statistical  analysis  is  the  last  chance  for  bias  or  invalid 
evidence  to  be  obscured  in  highly  technical  problems  and  language. 
The  documentation  and  opportunity  for  alternative  analyses  suggested 
above  create  a  situation  which  would  minimize  the  possibility  that 
such  bias  or  invalid  evidence  would  escape  the  attention  of  the 
legislative  user. 

Step  7:   Format  for  Evaluation  Studies. 

Reports  of  evaluation  studies  should  be  written  in  a  concise 
format  which  exclusively  links  the  legislative  goal(s)  at  hand  to 
the  program  results. 

The  format  we  have  in  mind  would  be  more  similar  to  a  legal 
brief  than  to  the  format  in  which  evaluation  results  typically 
have  been  reported.   Cross-referencing  of  evidence  to  their  data 
base  is  essential. 

Findings  should  be  cross-referenced  to  the  specific  evidence — 
i.e.,  to  the  raw  data,  measures,  research  design,  and  statistical 
analysis  on-  which  they  are  based.   The  purpose  of  this  format  is 
to  present  evidence  concisely  in  such  a  way  that  its  relevance  to 
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the  Issue  and  its  credibility  can  be  examined  and  reviewed  by 
Consrress.   Thus  an  evaluation  report  to  Congress  should,,  comprise)  >. ' 
these  cross-referenced  sections: 

1.  the  statement  of  the  goals  (cross-reference  to  results) 
as  drafted  by  the  committee, 

2.  the  evidence  which  the  evaluator  argues  is  pertinent, 
credible  data  on  that  issue,  <f*  * 

3.  sections  documenting  the  analyzed  data,  the  analytic 
techniques  (i.e.,  statistical  tests,  etc.)  the  raw  data,  the  methods 
by  which  they  were  gotten,  and  other  appropriate  technical  details 
on  which  the  credibility  of  the  evidence  might  be  challenged. 

Procedures  for  using  evidence  in  this  format  are  described 

in  Appendix  A. 

Beware  of  Summaries: 

Although  summaries  are  often  suggested  as  a  solution  to 
Congress'  information  overload,  they  discourage  andoften  pre-empt 
Congress1  rights  and  responsibilities  to  examine  evidence  directly 
and  draw  its  own  conclusions.   Hence,  we  recommend  against  the 
summary  as  an  exclusive  vehicle  for  presenting  evidence  from  evalu- 
ation studies  to  Congressmen  and  their  staff. 
•  ***»•*«* 

Why  do  this?  The  practical  purpose  of  the  steps  we  have 
outlined  here  is  to  provide  sufficient  evidence  to  justify  and 
document 

1.  Committee  actions  at  authorization  stage 

2.  Individual  members'  floor  votes 

3.  Actions  of  Appropriations  and  Budget  Committees 

4.  Challenge  to  Executive  branch  budgets. 
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The  new  Congressional  budgeting  system  will  require  a  flow 
of  evidence  at  a  relatively  specific  level  and  in  readily  usable 
and  checkable  form.   Applying  the  various  types  of  evaluation 
methods  can  provide  that  evidence  if  organized  and  applied  as 
outlined  here. 
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APPENDIX  A 
AN  ALTERNATIVE  FOR  EVALUATION  REPORTING 

Congressmen  can  use  evaluations  to  arrive  at  Congressional 
decisions  or  to  review  executive  branch  proposals — especially  the 
Presidential  budget  and  its  documentation.   The  committee  hearing 
can  be  a  vehicle  for  transmitting  and  organizing  the  evidence 
provided  by  evaluation  studies.   The  actual  reports  of  evaluation 
studies  can  be  presented  in  a  concise,  cross-referenced  format, 
similar  to  simplified  legal  briefs.   Principals  and  steps  which  can 
be  taken  to  organize  hearings  to  provide  an  efficient  education  in 
the  facts  at  hand  on  any  specific  issue  will  be  outlined  below. 

Identification  of  issues.   Hearings  should  be  organized  around 
a  clear  statement  of  the  issue  which  is  of  legislative  interest. 
For  example,  the  Introduction  of  new  legislation  raises  the  issue 
of  the  nature  and  extent  of  needs  addressed  by  this  legislation. 
The  appropriate  evidence  can  be  gathered  by  needs  assessment  and 
policy  analysis — as  indicated  in  Table  1.   If  program  implementation 
or  service  delivery  is  the  issue,  evidence  should  be  reported  from 
process  evaluations.   Other  examples  may  be  inferred  from  Table  1. 
The  important  point  is  that  the  issue  be  defined  to  focus  the 
hearing  and  appropriate  type  of  evidence  be  available  when  needed. 

Present  evidence  for  and  against  the  issue.   Much  of  the 
fogginess  which  frustrates  users  of  evaluation  reports  is  often 
due  to  efforts  to  avoid  or  conceal  bias  in  the  study.   Of  course 
reports  should  be  objective  not  an  exercise  in  propaganda  or 
entirely  personal  opinion,  but  they  will  be  most  useful  if  organized 
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into  clear  cut  documentation  of  positive  or  negative  positions 
on  the  issue.   For  example,  if  the  implementation  of  a  program  is 
at  issue  you  need  to  have  evidence  for  and  against  the  points  of 
implementation  at  issue. 

If  evidence  in  testimony  is  systematically  keyed  to  different 
sides  of  an  issue,  that  evidence  should  be  examined  and  used  more 
readily  than  if  it  is  varied  along  lengthy  caveats.   Advanced 
categorization  of  the  evidence  in  relation  to  the  issue  can  enhance 
Congressmen's  responsibility  to  make  a  layman's  judgement  on  the 
validity,  relevance,  and  sufficiency  of  technical  evidence  for 
differing  positions. 

Setting  reporting  and  hearing  dates.   With  a  little  thought 
you  can  identify  advance  broad  issues  which  will  arise,  in  connec- 
tion with  new  bills,  existing  acts,  or  the  budgeting/appropriations 
process — outlined  in  Table  1 — and  arrange  to  develop  the  kind 
of  data  you  will  need  via  the  legislative  language  in  Table  2A  in 
the  congressional  veto  process.   You  must  also  specify  reporting 
dates  for  receipt  of  evaluation  briefs  which  will  coincide  with  the 
times  at  which  issues  will  be  most  salient.   Most  routine  but 
significant  issues  can  be  foreseen  accurately  enough  to  be  built 
into  a  committee  schedule. 

"Cross-examination"  questions  and  staff  review.   On  receipt 
of  an  evaluation  study  brief,  committee  staff  should  review  the 
brief  and  formulate  a  list  of  questions  for  circulation  to  members 
which  bear  on  the  relevance,  validity  and  sufficiency  of  the  evidence 
in  the  brief.   These  questions  should  be  circulated  to  members  in 
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advance  of  the  hearings,  along  with  each  copy  of  each  brief. 
Such  questions  should  be  directed  to  the  witness  testifying  the 
actual  hearing. 

The  hearing.   The  procedure  outlined  so  far  is  aimed  at 

<  providing  Congressmen  with  access  to  direct  evidence.   The  hearing 
should  be  a  forum  in  which  challenges  to  that  evidence  can  be 
raised  and  settled.   Hence,  the  hearing  should  be  conducted  with 
the  support  of  examination  of  witnesses  by  expert  consultants  on 
behalf  of  the  committee  if  necessary. 

Results  of  the  hearings:   Workable  reports.   The  product  of  a 
hearing  report  should  be  evidence  which  has  been  sifted  and  sorted 
by  the  committee  to  guide  floor  voting,    for  authorization 
committee  support,  for  the  work  of  the  appropriations,  budget  or 
tax  committees.   Such  committee  reports  can  be  a  data  base  for  mark-up 
decisions,  committee  decisions  on  Congressional  budget  resolutions, 
as  well  as  for  challenging  the  Presidential  budget's  responsibility. 
Appeals:   Floor  Votes  and  Subsequent  Committees.  The  action 

,.  of  a  committee  can  in  effect  be  appealed  by  reexamination  of  evi- 
dence,  or  by  providing  new  evidence  at  points  of  subsequent  action. 
For  example,  in  appropriation  committees,  budget  committee  or 

I  floor  voting. 

Finally,  it  should  be  noted  that  the  format  for  reporting 

i  evaluation  studies  should  suit  the  needs  of  the  committee.   In 
some  cases,  this  can  be  done  in  a  committee  hearing  context,  other 

1  times  in  informal  meetings.   In  any  case  the  major  consideration, 
assuming  a  valid  evaluation  has  been  completed,  is  that  the  results 
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be  reported  in  a  form  that  will  be  useful  to  committees  and  member:*, 
of  congress  in  their  legislative  deliberations. 
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APPENDIX  B 


SUMMARIES  OF  EVALUATION  TECHNIQUES 
(Being  Developed) 
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Preprogram  measurements  on  the  measuring  instruments  that  are 
to  be  used  to  define  program  effects  are  usually  needed.   Yet  it  is 
common  for  a  new  program  to  be  implemented  immediately,  before  the 
measuring  instruments  have  been  developed  or  applied.   Where  program 
effectiveness  is  to  be  measured  with  novel  measurement  devices  (and 
this  is  the  usual  situation)  and  where  the  program  is  a  pilot  program 
being  instituted  to  inform  Congress  about  the  desirability  of  a 
potential  National  Program,  the  legislation  might  well  specify  that 
program  implementation  may  be  deferred  until  the  preprogram  measure- 
ment activity  has  been  completed.   Legislation  might  allow  as  much 
as  one  year  for  measurement  development  and  the  full  scale  prepro- 
gram measurement  activity.   (Where  appropriate  field-tested  measures 
already  are  available,  a  six-month  period  for  organizing  the  mea- 
surement staff  and  executing  the  preprogram  measurement  activity 
would  usually  be  sufficient.) 

Untreated  comparison  groups  are  often  needed  to  make  sure  that 
the  oreproerram-postprogram  differences  are  not  due  to  other  events, 
such  as  inflation,  recession,  the  shutting-down  of  a  major  employer, 
unusual  weather  conditions,  increasing  familiarity  with  the  mea- 
suring instruments,  and  the  like.   If  the  measuring  instruments 
are  such  that  other  probable  events  (in  addition  to  the  new  program) 
are  likely  to  cause  changes,  then  the  legislation  should  consider 
specifying  that  one  half  or  more  of  the  measurement  effort  be 
devoted  to  making  measurements  on  comparison  groups  not  receiving 
the  treatment.   At  present,  most  evaluation  staffs  do  not  feel 
they  have  this  authority,  and  many  evaluations  lack  this  much 
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APPENDIX  C 


CONSIDERATIONS  IN  DESIGNING  EVALUATION 
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needed  comparison  base.   (If  the  program  is  expensive,  it  is  usually 
cheaper  to  increase  the  statistical  dependability  of  the  results  by 
hnvinp;  considerably  more  persons  or  other  units  in  the  comparison 
group  than  in  the  program  group.   The  common  practice  of  having  the 
comparison  group  smaller  than  the  program  group  is  neither  eco- 
nomically or  statistically  efficient.^ 

Uncontrolled  variation  in  the  treatment  being  studied  was 
seldom  a  serious  problem  in  the  scientific  laboratory  settings  in 
which  the  designs  now  applied  to  program  effectiveness  evaluation 
were  developed.   In  such  scientific  settings  it  is  taken  for  granted 
that  the  kind  of  treatment  being  evaluated  is  carefully  and  fully 
described,  is  sufficiently  simple  to  so  describe,  and  is  in  fact 
provided  as  described.   The  wise  user  of  program  evaluation  can 
make  none  of  these  assumptions  about  the  program  treatment  whose 
effectiveness  is  being  evaluated.   However  if  the  legislative  user 
has  taken  our  advice  in  Steps  -j  and  4  regarding  measurement  of  pro- 
gram delivery,  confounding  of  program  effectiveness  by  uncontrolled 
variation  in  program  services  can  be  controlled  in  one  of  the 
three  following  ways. 

1.  Make  no  deliberate  effort  to  control  variation  in  program 
services  at  the  time  they  occur,  but  to  require  that  measures  of 
program  services  be  used  as  post  hoc  controls  in  the  statistical 
analyses  of  program  effectiveness  data.   This  alternative  is 
slightly  better  than  nothing,  but  often  falters  on  technical  problems. 

2.  A  second  alternative  is  to  require  that  a  program  be  certified 
as  complying- with  the  measurable  definitions  of  the  program  obtained 
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in  previous  steps.   Consistent  compliance  of  program  services 
throughout  the  period  of  effectiveness  evaluation  should  be  docu- 
mented.  In  effect,  this  alternative  provides  that  what  is  to  be 
evaluated  for  effectiveness  has  in  fact  been  delivered  as  lntended-- 
an  easy  assumption  in  a  laboratory,  but  not  in  the  real  world  of 
Federal  programs. 

3.  Planned  variation  in  program  services  is  a  third — and 
probably  best — way  to  assure  that  the  delivery  of  program  treatment 
does  not  confound  evaluation  of  its  effectiveness.   In  planned 
variation  the  levels  and  types  of  program  services  are  deliberately 
and  systematically  varied  in  a  logical  scheme  which  provides  the 
evaluator  with  maximum  opportunity  to  detect  program  effectiveness 
and  associate  it  with  program  services.   Planned  variation  requires 
full  cooperation  and  coordination  of  those  administering  program 
services  with  program  evaluators,  but  the  benefits  and  additional 
knowledge  gained  thereby  easily  offset  the  extra  effort  required. 
Measurement  of  program  services  is  a  necessary  condition  for  planned 
variation,  as  for  the  other  alternatives.   Thus  we  strongly  recommend 
that  legislative  users  of  program  evaluations  insist  that  evalua- 
tions of  program  effectiveness  include  measurable  definitions  of 
program  services,  valid  measurements  for  those  services,  and  the 
best  possible  opportunity  for  evaluators  to  rule  out  uncontrolled 
variation  in  program  treatments  that  might  confound  program  effect- 
iveness . 
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Randomized  selection  procedures  for  admission  to  the  program 
are  often  desirable  to  increase  the  comparability  of  the  comparison 
croup.   For  example,  in  compensatory  evaluation  programs,  it 
frequently  turns  out  that  the  only  schools  that  will  volunteer  for 
the  tedious  comparison  measurements  are  high  morale  schools  that 
have  nothing  to  fear  from  having  their  educational  success  measured. 
Such  comparison  schools  tend  to  make  the  needy  program  schools 
look  bad  even  when  the  program  is  effective.   Legislation  might 
well  permit  a  procedure  whereby  a  lottery  was  used  to  select  program 
schools  and  comparison  schools  from  among  applicant  schools  that 
had  agreed  to  cooperate  no  matter  which  group  the  lottery  assigned 
them  to.   If  the  program  is  a  pilot  program,  or  even  an  operational 
program  that  is  underfunded,  the  use  of  randomization  procedures 
to  select  program  and  comparison  groups  is  a  fair  and  democratic 
way  of  allocating  scarce  resources,  and  in  addition  makes  program 
e valuations  optimally  interpretable. 

Randomization  of  program  participation  among  eligible  parti- 
cipants is  desirable  when,  as  is  usual,  the  evaluator  can  neither 
know  nor  control  all  the  characteristics  of  participants  which  could 
influence  program  effectiveness  or  the  outcome  measures.   The 
principle  of  randomization  provides  procedures  whereby  every  po- 
tential participant  has  an  equal  chance  of  receiving  program 
services.   Opportunity  for  randomization  will  permit  use  of  much 
stronger  evaluation  designs  by  the  evaluator,  and  thereby  provide 
legislators  with  much  better  evidence  for  oversight.   In  authorizing 
use  of  randomization  of  program  participation  among  those  eligible, 
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legislative  users  of  evaluation  should  of  course  be  watchful  to  see 
that  basic  rights  of  participants  are  not  compromised.   Usually, 
however,  randomization  procedures  serve  to  increase  equality  of 
opportunity  rather  than  to  restrict  freedom.   Coerced  participation 
in  a  program  is  undesirable  with  or  without  randomization. 

Evaluation  research  design  is  extremely  complex  and  requires 
many  technical  Judgments  which  cannot  be  made  responsibly  by  the 
layman  untrained  in  research  design.   Legislators,  however,  do  not 
meet  their  own  responsibility  by  merely  surrendering  the  whole 
matter  to  experts.   Legislators  have  a  responsibility  to  raise 
basic  questions  about  the  validity  of  the  evidence  they  will  be 
getting  in  the  program  effectiveness  evaluations  they  authorize 
for  use  in  oversight. 

Quantified  Eligibility  criteria  for  programs,  if  strictly 
applied,  make  possible  a  second-best  substitute  for  randomized 
selection.   Most  programs  that  are  intended  for  the  neediest  (or 
the  most  deserving)  are  in  practice  administered  in  ways  that 
preclude  any  interpretable  comparison  between  the  outcomes  of  those 
who  get  the  program  and  those  who  do  not.   If  instead,  the  admission 
criteria  can  be  specified  quantitatively  (as  in  terms  of  income 
level,  or  reading  comprehension  scores,  or  multiple  rating  scales 
used  by  admission  staffs,  or  explicit  composites  of  such  ingredients) 
if  the  program  is  then  given  to  those  scoring  most  eligible,  if 
outcome  measures  are  made  not  only  for  the  program  recipients  but 
also  for  unaccepted  applicants  beyond  the  cutting  point  and  if  the 
outcome  measures  are  analyzed  in  terms  of  the  preprogram  eligibility 


496 


scores,  then  a  very  useful  estimate  of  program  effectiveness  can 
be  made.   To  make  this  possible,  legislation  might  well  specify  that 
the  program  be  given  to  the  most  eligible  applicants  and  that 
eligibility  should  be  in  terms  of  a  quantitative  index  based  upon 
specified  factors.   Legislation  specifying  a  quantitative  eligi- 
bility dimension  but  not  a  specific  cut-off  score  is  most  useful. 
Especially  when  individuals  are  being  selected  (rather  than  schools, 
etc.)  a  specific  cut-off  score,  if  generally  known,  can  lead  po- 
tential applicants  to  fail  to  apply  or  to  falsify  their  status 
(earnings,  etc.),  both  effects  tending  to  invalidate  the  analysis. 

Administrative  records  are  often  highly  relevant  to  program 
evaluation.   There  are  available  ways  of  using  them  that  fully 
protect  the  privacy  of  the  individuals  involved.   It  will  often  be 
optimal  for  legislation  to  specifically  authorize  s"uch  appropriately 
safeguarded  uses.   It  will  rarely  be  desirable  for  the  legislation  to 
preclude  such  use.   Many  programs  have  among  their  goals  modifying 
factors  that  are  already  being  recorded  by  some  government  agency; 
earnings  of  the  poor  and  undereducated ,  claims  on  unemployment 
insurance,  school  achievement  scores.   For  such  programs  admini- 
strative records  for  program  groups  and  comparison  groups  have 
many  advantages.   They  can  be  much  less  expensive  than  to  obtain 
information  on  the  same  topic  through  interviews.   They  can  cross- 
validate  interview  findings.   They  avoid  the  differential  cooperation 
and  attrition  often  found  between  experimental  and  comparison  groups. 
They  allow  for  time-series,  that  is,  for  records  on  program  and 
comparison  groups  for  a  number  of  years  prior  to  the  program  and  a 
number  of  years  afterwards. 
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Such  time  series  are  often  convincing  evidence  of  program 
effectiveness  even  without  a  comparison  group.   With  a  comparison 
group,  they  make  for  the  strongest  evaluation  design  possible. 

Competitive  replication  with  variation  is  an  extremely  impor- 
tant source  of  validity  in  science  which  can  easily  be  lost  in 
large-scale  program  evaluation.   Often  It  would  be  better  to  have 
the  program  evaluated  by  numerous  smaller  scale  independent  evalu- 
ations  than  by  one  large-scale  effort.   At^  very  least,  legislators 
should  consider  splitting  all  major  evaluations  into  two,  each  to  be 
implemented  by  a  separate,  independent  research  team.   The  measure- 
ment procedures  would  have  the  same  goals  but  different  details. 
The  many  arbitrary  decisions  that  have  to  be  made  in  any  research 
effort  would  be  made  differently.   If  two  such  independent  evalu- 
ations were  to  come  up  with  the  same  recommendations,  the  policy 
implications  would  be  clear.   If  they  were  to  disagree,  Congress 
would  be  thereby  informed  as  to  the  equivocality  of  the  evaluations. 

Competitive  reanalysis  of  program  evaluation  data  is  another 
aid  to  objectivity  which  legislators  can  specifically  encourage, 
by  requiring  that  all  data  from  all  measures  collected  be  made 
available  for  reanalysis  by  other  persons  (subject  to  deletion  of 
personal  identifiers  and  other  protection  against  disclosure  of 
information  on  individuals).   This  issue  is  discussed  in  more  detail 
in  the  manual. 
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The  conduct  of  program  evaluation  research  has  grown  considerably  In  recent  years, 
in  response  to  explicit  legislative  mandate  as  well  as  executive  agency  directive.   How- 
ever criticism  abounds  about  the  utility  to  the  government  of  many  of  these  types  of  re- 
search.  This  report  discusses  some  of  the  factors  both  governmental  and  non-govern- 
mental which  contribute  to  this  situation.   It  also  identifies  recent  actions  taken  to 
remedy  the  discrepancy  between  the  promises  and  utility  of  evaluation  research.   Re- 
maining issues  of  possible  legislative  concern  are  also  listed. 
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PART  I.   INTRODUCTION 

Federal  expenditures  for  program  evaluation  are  estimated  to  have  risen  by  more 
Kan  500  percent  from  1969  to  1974,  from  $20  million  to  more  than  $130  million.   This 
ncrease  in  program  evaluation  activities  reflects  two  Important  trends.   The  first 
.s  a  shift  in  total  Federal  budget  expenditures  toward  greater  emphasis  on  human  re- 
sources programs,  from  approximately  $65  billion  in  1969  to  almost  $173  billion  in 
1976.  _1/  The  second  trend  consists  of  a  "many-fold  multiplication"  in  congressional 
lppropriations  for  evaluations  in  response  to  increasing  demands  from  the  Congress, 
:he  President,  and  executive  agencies  for  greater  accountability  and  evaluation  of 
che  resources  devoted  to  social  programs.   _2/   The  conduct  of  program  evaluation  is 
also  part  of  a  broader  trend  to  provide  decisionmakers  in  both  executive  and  legisla- 
tive branches  of  Government  with  objective  information  or  policy  advice  on  the  effects 
of  Federal  actions  and  programs,  preferably  before  large-scale  application.   _j/  Dr. 


_1/  Executive  Office  of  the  President.   The  United  States  Budget  in  Brief,  Fiscal 
Year  1976,  Washington,  D.C.,  U.S.  Govt.  Print.  Off.,  1973.   p.  7. 

]J     Mushkin,  Selma  J.   Evaluation:   Use  With  Caution.   Evaluation,  v.  1,  1973!   31. 
The  Senate  Agriculture  Committee's  needs  for  evaluation  especially  an  evaluation 
of  the  Farmer's  Home  Administration  are  described  in  James  W.  Giltmier.   Policy 
Formation  Through  Program  Evaluation  and  Systems  Analysis;   A  Congressional  View, 
mimeo,  20  pages,  paper  presented  before  Evaluation  Seminar,  (held  monthly  under 
Sen.  Brock's  authority),  Seminar  held  November  20,  1974,  U.S.  Capitol. 

2_l     Program  evaluation  activities  required  under  the  Congressional  Budget  and  Impound- 
ment Control  Act  are  described  below  in  this  Introduction,  In  addition,  The  Office 
of  Technology  Assessment,  established  under  P.L,  92-484,  October  13,  1972,  as  an 
information  and  analysis  office  attached  to  the  Congress,  will  "...provide  [the 
Congress  with]  early  indications  of  the  probable  beneficial  and  adverse  impacts 
of  the  applications  of  technology..."  and  other  relevant  information,  to  assist 
in  the  legislative  process.   [See:   U.S.  Congress.  Senate.   Committee  on  Rules 
and  Administration.   Subcommittee  on  Computer  Services,   Technology  Assessment 
for  the  Congress,  staff  study,  November  1,  1972,  92nd  Congress,  2nd  session, 
Washington,  D.C.  U.S.  Government  Printing  Office,  1972.   (Committee  Print,)] 
Judicial  Interpretation  of  the  National  Environmental  Policy  Act  of  1969,  now 
reflected  In  revised  guidelines  issued  by  the  Council  on  Environmental  Quality, 
requires  the  submission  of  102  impact  statements  on  a  wide  variety  of  major  Fed- 
eral activities  which  may  impact  on  the  "human  environment,"  broadly  defined. 
These  activities  Include  construction  of  facilities,  alterations  to  the  natural 
physical  environment,  funding  proposals,  and  administrative  actions,  as  well  as 
proposals  for  different  types  of.  Federally  supported  scientific  research  and 
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H.  Guyford  Stever,  Che  President's  Science  Adviser  estimates  that  $835  million  will 
be  obligated  for  social  R  and  D  programs  in  the  fiscal  year  1975.   In  fact  in  dis- 
cussing the  1975  budget  message  he  singled  out  for  special  consideration  as  social  ex 
periments  designed  to  generate  information  for  public  policymaking,  three  programs  in 
health  insurance,  income  maintenance,  and  housing  allowances,  with  obligations  es 
to  total  $33.9  million.  _lj 

Concurrent  with  increasing  activity  in  social  program  evaluation,  however,  there 
is  evidence  of  heightened  concern  by  Members  of  Congress,  agency  program  administra- 
tors, and  social  scientists,  with  inadequacies  in  program  evaluation  methodology,  pro- 
cedures for  data  collection,  procurement,  administration,  and  utility  of  program  eval- 
uation, especially  in  planning,  oversight,  and  budgeting  functions. 

The  history  of  Federal  "social  program  evaluation"  activities  has  never  been  fully 
chronicled.   Similarly,  there  is  no  detailed  analysis  of  the  merits  of  evaluation  ob- 
jectives, programs,  and  approaches  used  by  different  social  program-oriented  agencies. 
This  report  is  designed  to  provide  the  Congress  with  background  information  on  existing 
and  emerging  issues  of  concern  relating  to  evaluation.   Part  I  is  this  introduction. 
Part  II  deals  with  some  general  issues  in  program  evaluation:   its  objectives,  commu- 
nications between  researchers  and  program  managers ,  and  the  difficulties  of  using  eval- 
uation in  agency  management  procedures.   Part  III  treats  0MB  program  evaluation  activ- 
ities, proposals  for  interagency  evaluation  mechanisms,  and  activities  surrounding  im- 
provement of  legislative  branch  programs  evaluation  capabilities.   Part  IV  deals  with 
issues  relating  to  the  procurement  and  conduct  of  program  evaluation  research  from  the 
point  of  view  of  both  the  evaluation  researcher  and  the  government  user.   Part  V  deals 
with  methodological  issues  in  the  conduct  of  program  evaluation.   The  possible  issues 


development  which  may  result  in  development  of  technology  with  a  potential  impact 
on  the  environment.   (See:   Council  on  Environmental  Quality.   Preparation  of 
Environmental  Impact  Statements,  Guidelines.   Federal  Register,  v.  38,  no.  14, 
Part  II,  August  1,  1973). 
_4/  The  Science  Adviser,  "Federal  Budget  for  Science  and  Technology  -  FY  1975," 
February  4,  1974:   8. 
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it   legislative  concern  raised  by  information  presented  in  the  report  are  collated  in 
Jart  VI. 

The  term  "program  evaluation"  is  used  in  two  general  ways  in  the  Federal  governnent. 
k.t  a  very  aggregated  level,  the  term  denotes  a  process  of  assessing  cost  and  effective- 
ness to  ascertain  the  appropriate  funding  level  for  a  program  in  relation  to  other 
aational  goals.  At  a  less  aggregated  level,  the  term  refers  to  the  processes  of  eval- 
uating a  particular  agency's  program  to  determine  whether  and  to  what  extent  it  achieves 
its  purposes  (impact  evaluation),  and  whether  it  is  administered  efficiently  (process 
evaluation).   Decisionmakers  use  these  evaluations  in  deciding  whether  a  program  should 
be  continued,  modified,  terminated,  or  replaced  by  another  "more  beneficial  and  more 
effective"  program. 

The  term  "program  evaluation"  has  been  associated  with  any  activity  to  measure  the 
effectiveness  and  outcome  of  a  program.   However,  it  has  been  most  widely  associated  in 
recent  years  with  evaluation  of  major  social  programs,  such  as  educational  intervention 
strategies,  work  incentives,  and  juvenile  training.   In  this  respect,  evaluation,  ac- 
cording to  Wholey,  et.  al.,  provides  "objective  information  to  program  managers  and 
policy-makers  on  the  costs  and  effects  of  national  [social]  programs  and  local  projects, 
thereby  assisting  in  [promoting]  effective  management  and  efficient  allocation  of  lim- 
ited resources."  V 

Federal  agencies  have  developed  a  number  of  different  types  of  program  evaluation 
mechanisms.   The  Congress  mandates  some  evaluations,  which  are  conducted  at  stipulated 
funding  levels  and  with  mandated  techniques.   Some  agencies  initiate  evaluation  to  pro- 
vide information  for  program  planning  and  budget  justifications.   Effective  coordinated 
program  evaluation  offices  exist  in  some  agencies.   Other  agencies  have  not  organized 
their  program  evaluation  activities  formally. 


_5/  Wholey,  Joseph  S.,  et.  al.   Federal  Evaluation  Policy:   Analyzing  the  Effects  of 
Public  Programs.   Washington,  D.C.,  The  Urban  Institute  [1970]  pp.  19-20. 
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One  characteristic  is  common  to  almost  all  social  program  evaluations  that  is 

methodology.  Thus  according  to  Wholey,  et.  al. ,  "evaluation  is  research,  the  applica- 
tion of  the  scientific  method  ...  to  learn  what  happens  as  a  result  of  program  activi- 
ties." Evaluation  thus  includes: 

the  definition  of  program  objectives,  the  development  of  measures  of  progress 
toward  these  objectives,  the  assessment  of  what  difference  public  programs 
actually  make,  and  the  projection  of  what  reasonably  could  be  expected  if  the 
program  were  continued  or  expanded.   _6/ 

The  tools  and  techniques  of  program  evaluation  usually  incorporate  the  basic  techniques 

of  social  science  research  systematic  collection  of  evidence  from  a  representative 

sample  of  the  population;  translation  of  information  into  quantitative  terms;  and  anal- 
ysis of  the  information,  using  statistical  manipulation  and  reference  to  theoretical 
notions  about  human  behavior,  to  "draw  conclusions  about  the  effectiveness,  the  merit, 
the  success,  of  the  phenomenon  under  study."   7/ 

Much  of  the  current  criticisms  of  program  evaluation  research  relate  to  its  state- 
of-the-art  and  its  questionable  utility  to  the  government.  A  number  of  recent  or  on- 
going governmental  and  professional  activities  illustrate  these  concerns. 

The  need  to  improve  the  state-of-the-art  of  program  evaluation  is  underscored  by 
the  establishment  within  the  Research  Applied  to  National  Needs  program  (RANN) ,  of  the 
National  Science  Foundation,  of  a  $2.5  million  program  to  provide  "rigorous  examination" 
and  refinement  of  the  techniques  of  program  evaluation  and  policy-oriented  research  in 
the  fields  of  labor,  youth,  health,  and  criminal  justice.   8/ 

The  Science  and  Technology  Policy  Office  of  the  National  Science  Foundation  is  sup- 
porting three  research  projects  on  social  science  policy,  designed  in  part  to  improve 
the  conduct  and  use  of  program  evaluation.   These  studies  are: 


J_l     Weiss,  Carol  H.   Evaluation  Research:  Methods  for  Assessing  Program  Effectiveness. 
Englewood  Cliffs,  Prentice-Hall  [1972]  p.  2. 

8/   "NSF  Initiates  Programs  for  Evaluation  of  Policy-Related  Research  Concerned  with 
Human  Resources."  NSF  News  Release,  April  11,  1973.   (NSF  73-137.) 
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National  Academy  of  Sciences,  Study  of  Social  Research  and  Development  and  its 
Role  in  Policy  Making,  6  mos.,  $116,500; 

Social  Science  Research  Council,  Conference  on  Social  Experimentation,  12  mos., 
$23,100;  and 

University  of  Washington,  Social  Policy  Analysis  and  Research  in  Social  Policy- 
making, 14  mos.,  $50,000. 

Some  of  these  topics  were  addressed  during  two  conferences  held  at  Dartmouth  during 
the  summer  of  1974.   The  first,  a  conference  on  Social  Experimentation,  was  sponsored  by 
the  Social  Science  Research  Council  with  funding  from  the  National  Science  Foundation. 
The  second,  the  "Dartmouth/Organization  for  Economic  Co-Operation  and  Development  (OECD) , 
Seminar  on  Social  Research  and  Public  Policies,"  was  supported  with  funds  from  the  Ford 
Foundation,  Dartmouth,  and  the  OECD.  The  Fifth  Annual  Conference  of  the  National  Capi- 
tal Area  Chapter  of  the  American  Society  for  Public  Administration  had  a  session  devoted 
to  examining  the  conduct  and  utility  of  program  evaluation.   It  was  h'eld  on  November  14 
and  15,  1974  in  Washington,  D.C. 

Numerous  bills  were  introduced  during  the  93rd  Congress  to  incorporate  rigorous 
program  evaluation,  pilot  test,  and  cost-benefit  analysis  procedures  into  legislation 
in  order  to  improve  congressional  oversight  of  the  budget.   The  Congressional  Budget  and 
Impoundment  Control  Act  of  1974  (P.L.  93-344,  approved  July  12,  1974),  established  a 
Program  Evaluation  Office  in  the  General  Accounting  Office  and  gives  the  agency  added 
responsibility  for  program  evaluation.   It  also  strengthens  legislative  responsibility 
for  using  program  evaluation  information. 

The  topic  of  assessing  the  quality  of  evaluation  research,  especially  experimenta- 
tion, for  decisionmaking  processes  is  also  receiving  some  attention  by  the  Special  Com- 
mittee on  Federal  Agency  Evaluation  Research  of  the  Division  of  Behavioral  Sciences  of 
the  National  Research  Council.   The  Office  of  Management  and  Budget  has  also  established 
an  Office  of  Evaluation  and  Program  Implementation  to  assess  these  issues.   In  addition, 
the  General  Accounting  Office  has  published  several  reports  directed  to  congressional 
committee  chairmen,  recommending  steps  to  be  taken  to  refine  legislative  language 
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requiring  program  evaluation.   The  GAO  is  now  preparing  a  guide  for  program  evaluation 
standards,  expected  to  be  released  in  the  Spring  of  1975. 

An  effort  is  made  in  this  report  to  present  background  information  in  a  manner 
which  serves  the  interests  of  the  Congress.  Many  of  the  issues  raised  may  be  relevant 
to  both  House  and  Senate  Budget  Committees  and  the  Congressional  Budget  Office,  estab- 
lished under  the  Congressional  Budget  and  Impoundment  Control  Act  of  1974,  P.L.  93-344 
of  July  12,  1974.  In  addition  some  authorizing  committees  as  well  as  committees  re- 
sponsible more  generally  for  overseeing  the  organization  and  administration  of  Govern- 
ment might  possibly  be  concerned  with  the  issues  raised. 

The  following  format  is  used  in  presenting  the  material.   Each  issue  is  developed 
by  referring  to  current  activities  and  literature,  such  as  those  of  the  Office  of  Manage- 
ment and  Budget,  the  General  Accounting  Office,  the  National  Science  Foundation,  and  the 
National  Academy  of  Sciences;  social  science  critiques;  and  published"  and  unpublished 
Federal  agency  reports.   Also  drawn  upon  if  applicable,  are  legislative  activities  which 
address  these  topics.   If  available,  recommendations  for  improvement  from  published 
sources  are  given.   The  possible  issues  of  legislative  concern  generated  by  the  report 
are  presented  in  Part  VI. 

(It  should  be  noted  that  the  Library  Services  Division  of  the  Congressional  Research 
Service  has  published  multilithed  annotated  bibliographies  on  program  evaluation.  All 
were  prepared  by  Nancy  Davenport.  They  a:e:  Evaluation  Research  in  Public  Administra- 
tion: Selected  References,  1967-1974,  October  21,  1974,. 9  p.;  Evaluations  of  Manpower 
Training  Programs:  Selected  References,  1970-1974,  October  25,  1974,  8  p.;  and  Evalua- 
tion Research  in  Social  Policy:   Selected  References,  1970-1974,  October  31,  1974,  8  p.) 
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PART  II.   THE  INITIATION  AND  USE  OF  PROGRAM  EVALUATION 

The  General  Accounting  Office  (GAO) ,  estimates  that  "Federal  expenditures  for  non- 
defense  program  evaluation  have  risen  dramatically  in  the  past  few  years  from  less 

than  $20  million  in  the  fiscal  year  1969  to  at  least  $110  million  in  fiscal  year  1972  _9_/ 
and  $130  million  in  the  fiscal  year  1974.   10/  As  a  result,  according  to  the  GAO: 

In  response  [to  the  increasing  demand  for  program  evaluation]  executive  agencies 
have  set  up  new  evaluation  offices  and  expanded  existing  offices.   Contracting 
for  evaluation  studies  has  expanded  greatly.   The  trend  in  number  and  size  of 
evaluation  studies  sponsored  by  the  Federal  Government  has  been  upward.   Private 
foundations  as  well  have  stepped  up  funding  of  evaluation  studies.   State  and 
local  governments  have  increased  evaluation  efforts  and  several  universities  have 
set  up  evaluation  institutions.  11/ 

Although  program  evaluation  is  extensively  used,  there  is  widespread  criticism  of 
its  utility  to  decision-makers.   For  instance,  Sen.  William  V.  Roth,  Jr.,  reporting  to 
the  Congress  on  a  1972  staff  study  entitled  Report  on  Survey  of  Federal  Program  Evalua- 
tion Practices,  said  "the  study  conducted  by  my  staff  suggests  serious  weaknesses  in 
agency  evaluative  and  analytical  procedures."  The  report,  he  continued,  "call[s]  at- 
tention to  the  need  for  the  executive  branch  to  improve  and  extend  its  attempts  to 


9 /  U.S.  General  Accounting  Office.   Program  Evaluation:   Legislative  Language  and  a 
User's  Guide  to  Selected  Sources.   Washington,  D.C.,  U.S.  General  Accounting  Of- 
fice [June  1973]  p.  1.   It  should  be  noted  that  most  of  the  literature  on  program 
evaluation  indicates  increasing  demand  by  both  Federal  agencies  and  the  Congress, 
and  increased  funding  by  agencies  to  support  the  conduct  of  program  evaluations. 
Dr.  Allen  Schick  disagrees.   He  reports  in  a  1973  publication:   "...There  has  been 
a  definite  drop  of  interest  in  large  program  evaluation,  though  many  Federal  agencies 
are  trying  to  institutionalize  evaluation  through  the  collection  of  output  and  per- 
formance data."   (Allen  Schick  (Senior  Specialist,  Congressional  Research  Service, 
Library  of  Congress) ,•  "The  Pilot  Testing  of  New  Programs:   An  Analysis  of  Title  IV, 
S.  3984  (92nd  Congress),"   In  U.S.  Congress.   Joint  Economic  Committee.   Subcom- 
mittee on  Budgeting,  Management,  and  Expenditures.   Improving  Congressional  Con- 
trol Over  the  Budget:   A  Compendium  of  Materials,  March  27,  1973.   Washington, 
D.C.,  U.S.  Govt.  Print.  Off.,  1973.   p.  276.   (Committee  Print.) 

10/  "Program  Evaluation  in  the  Executive  Branch:  Estimate  of  Funds  Budgeted  for  Fis- 
cal 1974,"  in  U.S.  Congress.  Joint  Committee  on  Congressional  Operations.  Con- 
gressional Research  Support  and  Information  Services:  A  Compendium  of  Materials. 
Washington,  U.S.  Govt.  Print.  Off.  [May  1974]   pp.  289-290. 

1]7  Program  Evaluation:   Legislative  Language  and  a  User's  Guide  to  Selected  Sources, 
op.  cit.:   p.  1. 
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measure  the  accomplishments  of  governmental  activities  and  weigh  these  accomplishments 
against  their  costs."   12/  Federal  administrators  who  fund  and  oversee  program  evalua- 
tion research  have  also  criticized  its  utility.  For  instance,  Robert  G.  Bruce,  Assistant 
Administrator  for  Program  Planning  and  Evaluation,  Social  and  Rehabilitation  Service, 
HEW,  reports:   "We  might  as  well  be  candid:   Federal  program  evaluations  so  far  have 
been  largely  ineffective."  12/     Garth  Buchanan  and  Joseph  Wholey,  program  evaluation 
researchers  at  the  Urban  Institute,  a  research  and  evaluation  agency  created  first  to 
serve  the  Department  of  Housing  and  Urban  Development,  observe  similarly: 

.  .  .  While  we  have  seen  that  increasing  amounts  of  money  are  being  budgeted  for 
evaluation,  we  conclude  that  this  increasing  support  must  be  due  to  a  continuing 
recognition  by  the  Government  thai  evaluation  information  is  needed  for  the  devel- 
opment and  management  of  social  programs,  rather  than  to  the  recognition  that 
evaluation  as  currently  practiced  is  the  answer  to  these  needs.  We  are  led  to 
this  conclusion  because  in  our  judgment,  the  impact  of  evaluation  results  on  pro- 
gram development  and  improvement  over  the  last  two  years  has  been  disappointing 
when  compared  with  the  amount  of  money  and  effort  that  has  gone  into  evaluation,  j^/ 

The  Office  of  Management  and  Budget  recently  initiated  a  major  effort  to  improve 

Federal  agency  program  evaluation  capabilities.  Agency  spokesmen  identify  four  basic 

difficulties  with  current  Federal  program  evaluation  research.  Their  concerns  parallel 

those  raised  above.  They  are:   (1)  timeliness  for  decisionmaking;  (2)  reliability  of 

the  data  used  and  the  methods  employed;  (3)  relevance  of  evaluation  activities  to  major 

policymaking  needs;  and  (4)  insuring  that  program  evaluation  reports  are  made  available 

to  and  utilized  by  decisionmakers.  _15/ 


12/  Public  Program  Analysis  and  Evaluation  for  the  Purposes  of  the  Executive  and  the 

Congress.   Statement  of  the  Hon.  William  V.  Roth  on  the  Floor  of  the  Senate.   Con- 
gressional Record,  June  8,  1972  (daily  edition):   S.  9026.   The  study  referred  to 
is:   Report  on  Survey  of  Federal  Program  Evaluation  Practices  Conducted  by  the  Staff 
of  Senator  William  V.  Roth,  Jr.,  (Delaware),  May  1972:   86  pp.   (Available  from 
Senator  Roth's  office.) 

!3/  Bruce,  Robert  G.  What  Goes  Wrong  with  Evaluation  and  How  to  Prevent  It.  Human 
Needs.  (Publication  of  Office  of  the  Administrator,  Social  and  Rehabilitation 
Service,  Department  of  Health,  Education  and  Welfare.)  v. I,  no.  1,  1972. 

!4  /  Buchanan, v Garth  N.  and  Joseph  S.  Wholey.   Federal  Level  Evaluation,   Evaluation  Fall 
1972:   21-22.   Weiss  holds  similarly  that  much  evaluation  research  falls  short  of 
the  expectations  of  funders,  program  staff  and  evaluators.   See  her  current  arti- 
cle: Weiss,  Carol  H.   Between  the  Cup  and  the  Lip  ...  Evaluation,  v. I.,  no.  1, 
1973:   49-55. 

15  /  Lewis,  Frank  L.  and  Frank  G.  Zarb.   Federal  Program  Evaluation  From  the  0MB  Per- 
spective.  Public  Administration  Review,  n.  4,  July/August  1974:   314. 
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The  literature  on  improving  the  administration  of  program  evaluation  raises  im- 
portant issues  which  may  warrant  additional  legislative  concern.   These,  which  will  be 
, developed  below,  are  to:   clarify  the  purposes  of  evaluation;  enhance  communication  be- 
tween administrators  and  program  evaluators;  integrate  program  evaluation  into  program 
management;  and  establish  better  information  dissemination  systems. 

A.   The  Need  to  Clarify  Legislative  Statements  of  Program  Goals  and  the  Purposes  of 
Evaluation. 

According  to  the  literature,  much  of  the  failure  of  program  evaluation  can  be  at- 
tributed to  policymakers  and  legislators  who  write  vague  statements  of  program  goals, 
which  give  evaluators  imprecise  yardsticks  or  standards  against  which  to  measure  pro- 
gram accomplishments  and  deficiencies  or  costs  and  benefits.   Dr.  Bruce  succinctly  char- 
acterized this  lack  of  definition: 

Program  goals  have  to  be  carefully  defined.   There  must  be  clear  statements  of 
what  a  program  is  supposed  to  accomplish  —  clear  in  the  sense  that  you  can  pick 
them  up  and  go  out  and  measure  whether  or  not  you  have  achieved  the  program  goals. 
Too  often  there  is  a  tendency  to  rely  upon  broad  statements  of  legislative  intent 
to  describe  the  purpose  of  a  program.   Such  statements  only  provide  a  context  for 
defining  specific  and  measurable  program  outcomes.   A  lack  of  such  agreed  upon 
definitions  to  serve  as  criteria  for  evaluating  a  program's  performance  may  be 
the  largest  single  cause  of  ineffective  evaluation  studies.   16/ 

1.   Discrepancies  Between  the  Purposes  of  Evaluation  and  the  Techniques  of  Evalua- 
tion.  The  literature  reflects  several  variants  on  this  theme  of  the  need  to  specify 
program  objectives  and  legislative  requirements  for  program  evaluation. 

One  these  receiving  attention  is:   many  programs  are  not  amenable  to  evaluation  be- 
cause of  the  lack  of  correspondence  between  their  broad  objectives  and  the  tools  and 
,  techniques  of  evaluation  research.   On  this  point,  Thomas  Morehouse  reports  that  Federal 
social  services  programs  take  one  of  two  forms:   maintenance  programs  or  opportunity 
programs.   A  maintenance  program  "provides  tangible  goods  or  services  to  a  clearly  de- 
fined population,  such  as  food  or  money.   An  opportunity  program,  in  contrast,  does  not 
directly  meet  the  material  needs  of  a  population,  but  is  designed  to  "increase  the  group's 

}^J     Bruce,  op.  cit. 
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capabilities  or  opportunities  to  acquire  goods  and  services  (and  sometimes  status  and 
power)  for  themselves."  These  opportunities  include  providing  the  individual  with  job 
training  or  education  to  give  him  the  skills  needed  to  obtain  directly  goods  and  ser- 
vices.  Innovative  social  programs,  «,n  which  most  program  evaluations  are  required, 
are  designed  to  provide  opportunity,  not  goods  or  services.   "The  consequence  for 
evaluation,"  Morehouse  reports,  "is  that  program  'outputs'  are  often  more  elusive  and 
less  easily  measured  than  the  outputs  of  tangible  maintenance  benefit  programs."  In- 
stead, these  programs  are  "explorations  of  problems,  objectives,  and  means."  Evaluators 
therefore  are  forced  to  evaluate  program  performance  by  using  input  measures,  such  as 
manpower  allocated  to  administer  the  program  or  funds  spent.   Such  techniques  yield  very 
little  information  about  the  effects  of  the  intervention  strategy  underlying  the  pro- 
gram.  Morehouse  concludes  that  social  scientists  who  conduct  evaluations  are  criticized 
by  policymakers  and  administrators  who  "continue  to  call  for   'objective  information' 
and  demand  'rigorous  scientific'  evaluation,"  and  by  fellow  social  scientists  who  call 
program  evaluation  research  trivia.   17/ 

A  second  widely  addressed  theme  is  that  non-specific  or  non-quantifiable  state- 
ments of  legislative  program  objectives  militate  against  evaluation  because  broad-ganged 
programs  cannot  be  evaluated  easily  with  existing  social  science  evaluation  techniques 
which  are  oriented  toward  experimentation.   18/  The  experimental  approach,  Morehouse 
notes,  requires  that  the  following  criteria  be  met:   "it  assumes  that  a  program  has  well- 
defined  objectives  and  that  its  effectiveness  can  be  determined  by  measuring  the  extent 
to  which  objectives  are  achieved;  the  setting  in  which  the  program  operates  should  be 
reasonably  "controlled";  the  program  "treatment  should  be  reasonably  uniform;  and  the 


17/  He  adds  that  this  is  true  "...even  if  performance  (using  experimentation  and 

quantitative  techniques)  has  often  fallen  far  short  of  aspirations,  and  even  if 
many  evaluation  research  projects  have  been  carried  out  in  other,  less  highly  re- 
garded ways."   (Morehouse,  871.)   The  basic  source  is  Thomas  A.  Morehouse.  Program 
Evaluation:   Social  Research  Versus  Public  Policy.   Public  Administration  Review 
[November/December  1972]:   872. 

/  For  "a  explanation  of  social  experimentation  and  its  use  in  social  irop-ii  evalua- 
tion research,  see  Part  V  of  this  report. 
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program  treatment  should  be  applied  In  a  large  enough  number  of  cases  to  provide  an  ade- 
quate sample  of  program  experience."   19/  However,  "most  of  the  Federal  social  action 
and  economic  development  programs  to  which  evaluation  requirements  have  been  attached 
do  not  fit  this  mold."  He  continues: 

[They  often  have]  broad  aims  that  cannot  be  specified  in  clear-cut  form;  they  may 
be  concerned  more  with  changing  a  situation  than  with  affecting  a  large  number 
of  individuals  directly;  they  generally  operate  in  open,  uncontrolled  community 
and  regional  settings;  and  a  given  program  tends  to  take  different  forms  in  dif- 
ferent situations.   The  programs,  in  short,  are  not  themselves  designed  or  ad- 
ministered as  experiments.   Thus  it  is  rarely  that  even  the  basic  data  collection 
requirements  for  evaluation  research  are  structured  into  program  operations,  let 
alone  the  requirements  for  standardized  treatment,  control-comparison,  and  repre- 
sentative samples.   20/ 

2.  Recommendations  to  Specify  Program  Goals  in  Legislation.  Most  evaluation  in 
recent  years  constitutes  a  response  to  legislative  requirements  incorporated  implicitly 
or  explicitly  into  authorizing  legislation.   Four  recent  reports  have  criticized  the 
vagueness  of  these  legislative  requirements  and  offered  suggestions  for  making  them  more 
specific,  to  conform  with  the  methods  and  procedures  of  current  evaluation  practices  and 
to  the  resources  authorized  to  conduct  evaluations .   21/ 

For  example,  in  March  of  1974,  the  General  Accounting  Office  provided  the  Congress 
with  an  inventory  and  analysis  of  legislative  language  requiring  program  evaluation. 
This  report,  and  an  earlier  one  submitted  in  August  of  1972,  were  designed  to  provide 
GAO  with  information  to  carry  out  its  expanded  evaluation  responsibilities  under  the 
Legislative  Reorganization  Act  of  1970.   22/  They  were  intended  also  to  assist  congres- 
sional committees  in  carrying  out  legislative  analysis  functions  required  under  section 


19/  Idem 

20/  Ibid.,  pp.  871-872. 

'21/  GAO  reports  cited  below;  Wholey,  et.  al.  Federal  Evaluation  Policy,  op.  cit.:  57- 
58;  and  National  Advisory  Council  on  Education  Professions  Development.   Search 
For  Success:   Toward  Policy  on  Educational  Evaluation.  A  Report  of  the  National 
Advisory  Council  on  Education  Professions  Development,  Washington,  U.S.  Govt.  Print. 
Off.,  June  1974.  pp.  35-36. 

22/  Legislative  References  to  Evaluation,  1967  (62  pages);  preceded  by  a  Letter  from 

the  Comptroller  General  to  the  Chairmen  of  each  Congressional  Committee,  August  11, 
1972.   (GAO  No.  B-161740)  ;  and  "Updated  Part  II  of  Program  Evaluation:   Legisla- 
tive Language,"  in  Congressional  Research  Support  and  Information  Services,  op. 
cit.,  pp.  338-339. 


514 


136  of  the  Legislative  Reorganization  Act  of  19A6,  amended  in  1970.  23/     The  General 
Accounting  Office  surveyed  legislative  requirements  for  program  evaluation  assessing 
their  wording  along  several  dimensions:   official  responsible  for  evaluation,  measures 
of  effectiveness,  methods  required,  report  recipients,  report  dates,  nature  of  recom- 
mendations, and  funds  authorized.   The  following  conclusions  were  derived: 

(1)  Of  the  40  acts  including  evaluation  references: 
15  acts  are  in  the  health  and  safety  area; 

5  acts  are  in  the  education  area; 
4  acts  are  in  the  transportation  area; 
3  acts  are  in  the  law  enforcement  area; 
2  acts  are  in  the  housing  area; 
2  acts  are  in  the  environmental  area; 
2  acts  are  in  the  international  area; 
2  acts  are  in  the  agricultural  area; 
2  acts  are  in  the  economic  opportunity  area; 

the  remaining  acts  are  each  in  a  different  area-aging,  selective 
service,  and  employment. 

(2)  Multiple  references  to  evaluation  (more  than  one  section  on  evaluation) 
are  included  in  19  of  the  40  acts. 

(3)  Specific  authorization  for  funding  evaluation  is  included  in  only  6  of 
the  40  acts. 

(4)  Specific  wording  concerning  measures  of  effectiveness  is  included  in  24 
acts. 

(5)  Eight  acts  include  language  specifying  methods  of  data  collection  and 
analysis.  However,  each  title  of  the  Economic  Opportunity  Act  includes  very 
specific  and  detailed  language  on  methods  of  evaluation  including  cost-benefit 
analysis,  use  of  control  groups,  and  standards  for  evaluation. 

(6)  Specific  reporting  date(s)  for  evaluation  results  are  included  in  36 
acts.  Reporting  requirements  range  from  a  very  general  time  span  (annually)  to 
a  very  specific  date  (on  or  before  January  31,  for  example) . 

(7)  There  is  a  wide  range  concerning  who  is  specified  as  responsible  for  con- 
ducting evaluations  and  who  is  specified  as  the  recipient  of  the  evaluation  reports. 
Most  of  the  laws  (and  sections  within  laws)  include  evaluation  references  specifying 
responsibility  for  conducting  evaluation  and  dissemination  of  results. 

(8)  Most  of  the  laws  in  the  health  area  and  many  of  the  laws  in  the  other 
areas  include  wording  for  recommendations  expected  from  the  evaluations.  Most 
recommendation  language  is  quite  specific.  Recommendations  are  expected  on  a 
broad  range  of  topics  including  changes  in  the  legislation,  changes  in  the  program, 
changes  in  program  plans,  and  improvement  in  evaluation  methods  and  measures.  24 / 


2  3'  According  to  GAO  this  section  "directs  standing  committees  of  the  House  and  Senate 
to  review  on  a  continuing  basis,  the  application,  administration,  and  execution  of  ■ 
those  laws,  or  parts  of  laws,  within  its  jurisdiction  in  order  to  assist  the  House 
and  Senate  in  (1)  its  analysis,  appraisal,  and  evaluation  of  the  application,  ad- 
ministration, and  execution  of  the  laws  enacted  by  the  Congress,  and  (2)  its  formu-- 
lation,  consideration,  and  enactment  of  such  modifications  or  changes  in  those  laws, 
and  of  such  additional  legislation  as  may  be  necessary  or  appropriate." 

2 4/  Congressional  Research  Support  and  Information  Services,  op.  cit.,  pp.  338-9.   The 
♦0  acts  requiring  evaluation  are  summarized  in  the  publication. 


515 


CRS  -  13 

In  support  of  assisting  committee  chairmen's  efforts  to  improve  the  quality  of 

legislative  language  requiring  evaluation,  the  GAO  subsequently  suggested  the  following 

"model"  of  statutory  evaluation  requirements: 

The  (head  of  the  agency)  shall  submit  an  evaluation  report,  to  (committees 
on  appropriations  and  committees  having  legislative  jurisdiction  over  the  pro- 
gram) no  later  than  of  each  year.   Such  report  shall — 

1.  contain  the  agency's  statement  of  specific  and  detailed  objectives  for 
the  program  or  programs  assisted  under  the  provisions  of  this  Act,  and 
relate  these  objectives  to  those  in  this  Act. 

2.  include  statements  of  the  agency's  conclusions  as  to  effectiveness  of 
the  program  or  programs  in  meeting  the  stated  objectives,  measured 
through  the  end  of  the  preceding  fiscal  year. 

3.  make  recommendations  with  respect  to  any  changes  or  additional  legisla- 
tive action  deemed  necessary  or  desirable  in  carrying  out  the  program 
or  programs . 

4.  contain  a  listing  identifying  the  principal  models,  analyses  and  studies 
supporting  the  major  conclusions  and  recommendations,  and 

5.  contain  the  agency's  annual  evaluation  plan  for  the  program  or  programs 
through  the  ensuing  fiscal  year  for  which  the  budget  was  transmitted  to 
Congress  by  the  President.   25/ 

3.   Criticisms  of  Specifying  Program  Goals  in  Legislation.   It  should  be  noted  that 
important  objections  have  been  raised  to  the  notion  of  the  Congress's  specifying  ex- 
plicit legislative  program  objectives  and  goals  for  program  evaluation  purposes.   Dr. 
Harold  Orlans,  of  the  Brookings  Institution,  reports  that  complex  political  and  plu- 
ralistic forces  in  the  legislative  process  generate  "...social  programs  [which]  are 

marked  by  . . .  multiple  and  contending  forces Congress  must  serve  more  easily  by 

,  amalgam  and  compromise  measures,  which  offer  something  to  many  different  groups  and 
1  are  lent  a  semblance  of  unity  by  goals  broad  enough  to  embrace  the  spectrum  of  interests 
1  involved."  26/  He  substantiates  this  notion,  in  part,  with  comments  from  Charles 
Schultze,  based  on  his  experiences  as  the  Budget  Bureau  Director  responsible  for  im- 
plementing the  Programming-Planning-Budgeting  System: 


25/   Marvin,  Keith  and  James  L.  Hedrick.  GAO  Helps  Congress  Evaluate  Programs.  Public 
Administration  Review,  no.  4,  July/August  1974:   331. 

26/  Dr.  Orlans  bases  his  comments  on  a  study  he  prepared  on  "The  Use  of  Social  Re- 
search in  Federal  Domestic  Programs,"  prepared  for  the  Research  and  Technical 
Programs  Subcommittee  of  the  House  Committee  on  Government  Operations.   They  are 
in:   Contracting  for  Knowledge:   Values  and  Limitation  of  Social  Science  Research, 
Jossey-Bass,  Inc.,  1973.   pp.  123-124. 
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"[Schultze]  points  out  how  diametrically  opposed  are  the  evaluator's  need  for 

a  clear  statement  of  program  goals  and  the  contrary  need  for  vagueness  in  getting  a 

program  adopted  in  the  first  place: 

The  first  rule  of  the  successful  political  process  is,  'Don't  force  a 
specification  of  goals  or  ?nds.'   .  .  .  [The]  necessary  agreement  on  particular 
policies  can  often  be  secured  among  individuals  or  groups  who  hold  quite  diver- 
gent ends.   .  .  .The  Elementary  and  Secondary  Education  Act  of  1965  .  .  .  was 
enacted  precisely  because  it  was  constructed  to  attract  the  support  of  three 
groups,  each  with  quite  different  ends  in  view.   Some  saw  it  as  the  beginning 
of  a  large  program  of  Federal  aid  to  public  education.   The  parochial  school 
interests  saw  it  as  the  first  step  in  providing  .  .  .  financial  assistance  for 
parochial  school  children.   The  third  group  saw  it  as  an  antipoverty  measure.  . 
If  there  had  been  any  attempt  to  secure  advance  agreement  on  a  set  of  long- 
run  objectives,  important  elements  of  support  for  the  bill  would  have  been 
lost,  and  its  defeat  assured.   27/ 

As  a  variant  of  this  theme,  the  Research  and  Policy  Committee  of  the  Committee 
for  Economic  Development  in  a  report  on  Improving  Federal  Program  Performance,  notes 
that  program  objectives  and  legislative  statements  of  intent  are  so  complex  that  ade- 
quate output  or  achievement  measures  should  be  developed  only  after  the  program  has 
begun  and  enough  time  has  elapsed  to  indicate  its  impacts.   28/ 

B.   Better  Communication  for  Effective  Evaluation. 

Another  issue,  frequently  treated  in  the  literature,  is  the  need  for  improved 
communication  between  social  program  evaluators  and  agency  staff  who  develop  program 
objectives  and  requirements  for  evaluation,  in  order  to  enhance  the  correspondence 
between  the  capability  of  evaluators  and  the  expectations  for  evaluation.  Generally 
reports  on  the  need  for  better  collaboration  focus  on  the  following  themes: 


2  7/  Orlans,  op.  cit.,  pp.  124-5,  citing:   C.L.  Schultze  The  Politics  and  Economics 
of  Public  Spending.  Washington,  D.C.,  Brookings  Institution,  1968.   pp.  47-49. 

28/  Improving  Federal  Program  Performance.  A  statement  on  national  policy  by  the 

research  and  policy  committee  of  the  Committee  for  Economic  Development,  Septem- 
ber 1971:   p.  61. 
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-there  is  a  need  for  social  scientists  and  program  administrators  to  determine 
jointly  objectives  of  program  evaluation  and  methodologies  for  research  before 
an  RFP  is  published  ;  29/ 

-program  evaluators  who  are  social  scientists  frequently  prepare  long,  jargon- 
ladden  reports  which  are  not  written  in  a  manner  which  meets  policymaking  needs 
and  which  are  often  overdue,  causing  Federal  agencies  to  award  funds  for  program 
evaluation  to  profit-making  research  units  which  prepare  timely,  but  often  inade- 
quate and  sometimes  invalid  program  evaluations;   30/ 

-Many  social  scientists  shy  away  from  doing  program  evaluation  research  because 
they  do  not  consider  it  fruitful  for  generating  basic  information  and  knowledge 
required  to  advance  the  social  sciences  as  disciplines,  and  the  university  peer 
reward  structure  does  not  value  evaluation  research  the  way  it  does  basic  re- 
search and  applied  research  in  the  social  sciences; 

-Federal  agencies  are  not  properly  organized  to  administer  social  science  research 
programs,  and  administrators  do  not  understand  the  complexities  and  limitations 
of  social  science  analytical  capabilities;  and 


This  issue  is  explored  in  greater  detail  in  Part  IV  of  this  report.   However 
it  should  be  noted  that  literature  on  this  theme  is  both  vituperous  and  construc- 
tive.  Reflecting  the  first  position:   Behavior  Today,  a  newsletter  distributed 
to  the  social  science  community,  reports  the  following  on  discussion  at  a  ses- 
sion of  the  September  1971  convention  of  the  American  Sociological  Association: 

NSF's  Joel  Snow,  OE's  John  Evans  and  NIMH's  Kenneth  Lutterman  painted  a 
bright  future  for  social  science  funding,  but  emphasized  priority  on  ap- 
plied and  evaluation  research.   Evaluation  research,  suggested  Peter 
Rossi  from  Johns  Hopkins,  in  a  breezy  but  blunt  rebuttal,  often  amounts 
to  research  on  nonsense.   'Sociologists  ought  to  have  the  guts  to  say  so.' 
Politicians,  Rossi  claimed,  controlled  the  conception  of  the  poverty  pro- 
gram, the  community  health  centers,  which  he  called  silly,  and  NSF's  new 
Rann  program  which  he  dubbed  'a  WPA  for  engineers  and  political  scientists'. 
Sociologists,  he  explained  to  the  government  representatives,  are  hesitant 
to  spend  three  years  of  their  lives  evaluating  such  programs.   There's 
something  wrong  with  social  science  and  the  federal  government,'  said 
Rossi.   'We  never  get  a  chance  to  define  the  problem.  The  problems,  even 
the  social  scientists  themselves,  are  defined  by  the  reigning  politicians.' 

See  also:   Program  Evaluation:   Review  of  Major  Issues  and  Literature,  a 
report  prepared  with  the  Instructions  of  the  Hon.  Roman  L.  Hruska.   By  Genevieve 
J.  Knezo,  Science  Policy  Research  Division,  Congressional  Research  Service,  Li- 
brary of  Congress,  January  31,  1972:   247  pages.   (Reproduced  and  distributed  by 
Law  Enforcement  Assistance  Administration,  United  States  Department  of  Justice, 
May  19,  1972.) 

For  additional  information  on  this  issue,  see  Part  IV  of  this  report. 
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-program  evaluators  must  become  more  sensitive  to  the  political  considerations 
which  impact  on  the  initiation  and  use  of  program  evaluation;  they  should  con- 
sider evaluating  the  political  impacts  of  a  program  or  at  least  be  sensitive  t 
the  political  values  impacting  on  a  decisionmaker  who  may  or  may  not  use  the 
evaluation  report.  31  / 


31/       The  theme  of  political/bureaucratic  barriers  and  resistance  to  the  use  of 
evaluation  and  social  experiments  surfaced  repeatedly  during  the  Dartmouth/OECD 
Conference  on  Social  Research  and  Public  Policies,  September  1974.   In  summary 
it  was  noted  that  the  same  political  consensus  or  conditions  which  permit  the 
conduct  of  experiments  or  evaluations  produce  independent  political  consensus  to 
adopt  a  program  even  before  the  results  of  the  research  are  available.   Similarly, 
an  evaluation  or  experiment  which  generates  positive  results  for  the  intervention 
strategy  will  not  lead  to  large-scale  adoption  of  the  program  if  political  condi- 
tions do  not  permit  it.   Good  evaluations  which  yield  negative  results  for  pro- 
gram effectiveness  may  not  be  used  because  legislators  and  agency  program  managers 
don't  want  to  see  their  pet  project  cut.  Good  or  bad  evaluations  which  yield 
positive  results  for  program  effectiveness  may  spell  the  demise  of  a  program  since 
the  program  may  be  perceived  as  having  achieved  its  purposes . 

For  additional  information  on  these  topics  see:   Campbell,  Donald  T.  Assessing 
the  Impact  of  Planned  Social  Change,  Background  Paper  for  the  Dartmouth  OECD  Seminar 
on  Social  Research  and  Public  Policies,  September  13-15,  1974,  passim;  Weiss,  Carol 
H.  Where  Politics  and  Evaluation  Research  Meet.   Evaluation,  v.  1,  n.  3,  1973: 
37-45;  and  Boeckmann,  Margaret.   Policy  Impacts  of  the  New  Jersey  Income  Mainte- 
nance Experiment.   A  paper  presented  at  the  American  Sociological  Association 
Meetings,  Montreal,  August  25-29,  1974,  mimeo:   28  pp. 

An  in-depth  study  on  how  decisionmakers  use  and  do  not  use  scientific  and 
technical  information,  including  social  science  information,  is  being  conducted 
by  Nathan  Caplan  at  the  Institute  for  Social  Research  University  of  Michigan. 
Kor  a  short  description,  see:   Science  Is  Seldom  Put  to  Good  Use  by  U.S.  Officials. 
ISR  Newsletter,  v.  2.,  n.  1,  Spring  1974:   2,  8. 

Recommendations  for  social  scientists  to  devote  more  attention  to  the  broker 
concept  in  communications  —  to  obtain  a  better  awareness  of  policymakers'  needs 
and  modes  of  behavior,  decisionmaking  requirements,  and  constraints  may  be  found 
in:   Lane,  Robert.   Social  Science  Research  and  Public  Policy.   Policy  Studies 
Journal,  v.  1,  n.  2,  Winter  1972:   111,  Jones, Ernest.   Law,  Political  Science, 
and  Policy  Studies.   Policy  Studies  Journal,  Autumn  1973:   56-71;  Elkin  Stephen  L. 
Political  Science  and  the  Analysis  of  Public  Policy.   Public  Policy,  v.  22,  n.  3, 
Summer  1974:   399-422;  Shonfield,  Andrew.   The  Social  Sciences  in  the  Great  De- 
bate on  Science  Policy.   Minerva,  July  1972:   426-438;  Horowitz,  Irving  Louis. 
Social  Science  Mandarins:   Policymaking  as  a  Political  Formula.   Policy  Sciences, 
v.  1,  Fall  1970:   339-360;  and  Levin,  Martin  A.  and  Horst  D.  Dornbusch.   Pure  and 
Policy  Social  Science:   Evaluation  of  Policies  in  Criminal  Justice  and  Education. 
Public  Policy,  v.  21,  n.  3,  Summer  1973:   383. 
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C.   The  Integration  of  Program  Evaluation  Into  Program  Management. 

Recent  congressional  reviews  of  program  evaluation  in  specific  agencies  and  social 
science  assessments  of  ways  to  improve  the  use  of  program  evaluation  research  stress 
that  evaluation  would  be  more  useful  if  these  activities  were  better  integrated  into 
policy  planning  and  program  management.   It  is  asserted  that  social  scientists,  pro- 
gram evaluation  researchers,  and  program  evaluation  managers  must  take  steps  to  enhance 
this  cross-fertilization  of  the  conduct  and  utility  of  such  research. 

1.   Incorporation  of  Evaluation  Into  Agency  Program  Planning  and  Budgeting.   The 

absence  of  a  link  between  evaluation  research  and  decisionmaking,  especially  budgeting, 

is  probably  the  most  important  impediment  to  effective  use  of  evaluation  research. 

Buchanan,  Horst  and  Scanlon  recently  completed  an  examination  of  the  evaluation  and 

decisionmaking  systems  of  the  Department  of  Labor  and  the  National  Institute  of  Mental 

Health.   On  the  need  to  incorporate  evaluation  into  decisionmaking  in  these  agencies, 

and  by  implication  into  others,  they  observe: 

[An  organizational  adjustment  is  needed  to  establish]  a  formal  relationship 
between  evaluation  planning  and  an  agency's  decision  processes.   In  the  ab- 
sence of  such  a  relationship,  the  same  types  of  evaluation  questions  are 
raised  year  after  year  in  different  decision  contexts  —  issue  papers,  strat- 
egy papers,  and  legislative  proposals  —  without  any  apparent  contribution  or 
resolution  by  the  evaluation  system.   If  the  evaluation  information  is  ever  to 
penetrate  agency  operations,  the  evaluation  models  will  have  to  become  an  inte- 
gral part  of  the  decision  process.   This  will  require  training  of  key  staff  — 
evaluation  planners,  program  managers,  policy-makers  —  in  the  design  and  use 
of  the  evaluation  models.   32/ 

Buchanan  and  Wholey  observe  that  Federal  agencies  have  a  responsibility  for  in- 
corporating evaluation  into  program  management  systems  and  that  "the  establishment  of 
an  evaluation  system  usually  requires  the  integration  of  several  systems  relating  to 
planning,  management,  and  data  collection."  33_/   Such  integration  necessitates 


32 /  Buchanan,  Garth  Pamela  Horst,  and  John  Scanlon.   Improving  Federal  Evaluation 
Planning.   Evaluation,  v.  1,  no.  2,  1973:   90. 

33/  Buchanan,  Garth  N.  and  Joseph  S.  Wholey.   Federal  Level  Evaluation.   Evaluation, 
Fall  1972:   22  and  passim. 


71-867  O  -  76  -  34 
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"coordination  among  several  organizational  units  and  sometimes  ...  changes  in  the  ad- 
ministrative structures  themselves,"  actions  which  are  not  easily  taken  by  entrenched 
Washington  bureaucrats.  Walter  Williams,  a  former  chief  of  the  Research  and  Plans  Di- 
vision, Office  of  Economic  Opportunity,  evaluated  the  organizational  problems  surrounding 
the  failure  of  major  social  program  evaluations  of  HEW,  HUD,  the  Department  of  Labor 
and  the  Office  of  Economic  Opportunity  in  the  late  1960s.  _34/  He  made  the  following 
administrative  recommendations  to  fashion  closer  functional  authority  between  research 
and  implementation:   (paraphrase) 

(1)  the  central  research  analyst  should  be  given  more  power  to  generate  research 
requirements  and  policy  analysis  required  to  develop  recommendations  regarding  pro- 
gram funding  and  implementation; 

(2)  the  analyst  should  be  given  more  money  to  conduct  long-range  social  experi- 
mentation research  and  evaluation; 

(3)  more  feedback  loops  should  be  built  between  the  field  and  the  central  office 
to  overcome  some  problems  of  implementation; 

(4)  the  central  analyst  should  have  capability  to  fund  social  science  research 
which  leads  to  recommendations  on  program  improvement  rather  than  merely  program 
achievement;  and 

(5)  more  attention  should  be  paid  to  stressing  outcome  objectives  in  implementa- 
tion rather  than  input  measures.   35/ 


_3jty  Williams'  major  points  were  summarized  in  an  article  reviewing  major  recent  criti- 
ques of  program  evaluation  and  social  experimentation  methodology:   "(1)  the  major 
failure  of  social  action  programs  in  the  1960s  was  in  not  being  able  to  move  from 
the  design  of  the  program  to  the  implementation  of  the  program,  particularly  in 
the  field;  (2)  the  major  deficiency  in  data  was  that  it  was  macronagative   (showing 
how  many  blacks  were  below  the  white  age  group  in  certain  reading  achievement  tests) 
instead  of  micropositive  (how  do  you  improve  that  reading  level) ;  (3)  the  central 
analysis  focused  too  much  on  what  was  happening  in  Washington  and  not  enough  on 
what  was  happening  at  the  local  level;  and  (4)  the  social  science  research  community 
is  not  structured  to  respond  to  the  analytical  needs  of  social  agencies  -  its  re- 
ward structure  mitigates  against  research  directly  in  support  of  policy  analysis." 
(Robert  E.C.  Wegner.   Evaluating  the  Evaluators.   Public  Administration  Review 
[January /February  1973]:   85-6). 

35/  Williams,  Walter.  Social  Policy  Research  and  Analysis:  The  Experience  in  the 
Federal  Social  Agencies.  New  York,  American  Elsevier  Publishing  Company,  Inc. 
[1971]   pp.  180-1. 

Weiss 's  report  on  administrative  obstacles  to  effective  program  evaluation  in 
the  National  Institute  of  Mental  Health  concludes: 

"...While  not  all  of  the  impediments  can  be  planned  away  in  advance,  certain 
basic  [organizational]  conditions  should  be  present  before  evaluation  grants  are 
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2.  Tne  Development  of  Evaluation  Work  Plana.   A  second  class  of  problems  dis- 
cussed in  relation  to  incorporating  program  evaluation  into  management  is  the  need  for 
an  agency  to  develop  comprehensive  work  plans  for  evaluation.   This  includes  setting 
priorities  among  programs  to  be  evaluated  and  designing  a  succession  of  evaluation 
studies  which  provide  answers  required  to  determine  the  adequacy  of  the  program  over 
time.   Wholey,  et.  al.  summarized  this  problem  in  the  1970  study: 

With  few  exceptions,   Federal  agencies  have  not  had  adequate  work  plans  for 
evaluating  their  major  social  programs. 

The  lack  of  evaluation  work  plans  often  resulted  in  haphazard  letting  of 
unrelated  contracts  to  evaluate  the  same  program.   Contractors  designed  their 
own  studies,  formulated  their  own  questions,  and  naturally  produced  data  and 
findings  that  were  not  comparable  with  each  other.   36 / 

3.  LEAA:   An  Illustration  of  These  Deficiencies.   A  congressional  investigation 
of  the  administration  of  the  block  grant  program  of  the  Law  Enforcement  Assistance 
Administration  attributes  major  failures  in  program  planning  and  evaluation  to  the  ab- 
sence of  a  well-formulated  plan  for  evaluation.  Many  of  the  findings  of  this  study 
probably  also  characterize  the  social  program  evaluation  mechanisms  of  other  Federal 
agencies.   Because  of  their  generality  and  potential  contribution  to  additional  legis- 
lative inquiry  on  evaluation  aechanisas ,  these  findings  will  be  sunnarized: 

-Although  the  LEAA  has  authority  under  its  enabling  legislation  to  "conduct 
evaluation  studies  of  the  programs  and  activities  assisted  under  the  title," 
the  agency  has  done  little  toward  making  its  own  evaluation  of  the  effective- 
ness of  programs  or  projects  funded  with  block  grant  funds.  37/ 

-One  of  the  major  deficiencies  LEAA  encounters  in  developing  adequate  evaluation 
plans  is  that  because  of  poor  information  systems,  the  agency  lacks  knowledge 


made.   Among  the  most  important  are:   (1)  clarity  of  purpose  for  the  study,  (2) 

a  well-defined  and  relatively  stable  program,  (3)  administrative  support,  and 

(4)  an  able  research  staff  who  give  a  substantial  proportion  of  time  to  the  study." 

(Weiss,  Carol  H.  Between  the  Cup  and  the  Lip Evaluation,  v.  I,  no.  2,  1973: 

55.) 

36/  Wholey,  et.  al. ,  Federal  Evaluation  Policy,   op.  cit.   p.  35. 

37/  U.S.  Congress.   House  Committee  on  Government  Operations.   The  Block  Grant  Program 
of  the  Law  Enforcement  Assistance  Administration.   Part  1,  Hearings  before  a  Sub- 
committee of  .  .  .  July  1971.   Washington,  U.S.  Govt.  Print.  Off.,  1971.   p.  143. 
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about  the  disbursement  of  grant  funds  and  the  purposes  to  which  these  funds  will 
be  spent.  This  deficiency  should  be  rectified  with  an  improved  grant  management 
information  system.   38/ 

-Equally,  if  not  more  important  in  terms  of  evaluating  management  performance 
and  capabilities,  neither  LEAA  nor  State  agencies  which  administer  the  programs 
have  formulated  standards  for  evaluating  program  progress,  success,  or  failure. 
Thus,  the  programs  are  unevc-.^ated,  unaudited,  and  incapable  of  being  measured 
as  to  performance  and  progress  due  to  the  lack  of  goals  or  standards.   39/ 

-Since  LEAA  does  not  maintain  an  inventory  of  grant  programs  and  lacks  a  central- 
ized evaluation  plan  and  standard,  its  efforts  sometimes  duplicate  those  of  other 
agencies.  40/ 

-Many  LEAA  funds  are  administered  directly  by  State  Planning  Agencies,  (SPA). 
The  lack  of  program  goals  and  evaluation  standards  and  guidelines  prevents  SPAs 
from  adequately  administering  and  evaluating  programs.   LEAA  has  the  responsi- 
bility for  providing  SPAs  with  manuals  and  guidelines  for  evaluation  but  has 
not  adequately  responded  to  this  need.   41/ 

4.   The  Contribution  of  Evaluation  Work  Plans  to  Evaluations  Performed  at  the 
State  Level.   Federal  assistance  to  State  and  local  jurisdictions  for  developing  evalu- 
ation plans,  objectives,  methodologies,  and  information-dissemination  systems  seems  all 


38/   Specifically:   ".  .  .  .LEAA  [lacks]  knowledge  of  the  purposes,  goals,  and  specifi- 
cations of  the  thousands  of  State  and  local  projects  financed  by  its  block  grants. 
This  informational  void,  still  unfilled  as  LEAA  completes  its  fourth  year  of  opera- 
tion, makes  LEAA  powerless  to  modify  programs  to  avoid  deviations  and  to  follow 
congressional  intent.   A  grant  management  information  system  and  a  reference  ser- 
vice, both  of  which  should  alleviate  these  and  other  monitoring  problems  of  LEAA, 
are  currently  being  devised  by  outside  consultants  under  contract  to  the  agency." 
(U.S.  Congress.   House.   Committee  on  Government  Operations.   Block  Grant  Program 
of  The  Law  Enforcement  Assistance  Administration.   Twelfth  Report  by  the  Comm.  on 
Government  Operations.   House  Report.  No. 92-1072.   May  18,  1972.   Washington,  D.S. 
Govt.  Print.  Off.,  1972.   p.  8.) 

39/   Idem. 

40/  Ibid.,  p.  61. 

4V  Specifically:   Statement  of  Mr.  Ahart.   "In  the  SPA  Guide  for  1969  (a  manual  is- 
sued by  LEAA  to  the  State  Planning  agencies  for  guidance  on  application  awards, 
and  administration  of  planning  and  block  grants).   LEAA  stated  that  it  would  is-  . 
sue  guidelines  suggesting  appropriate  procedures,  techniques,  and  measures  for 
evaluating  the  contribution  to  crime  control  of  the  block  grant  projects  and  ex- 
penditures.  The  guide  provided  that  the  State  pi  arming  agencies,  pending  issuance, 
of  the  guidelines,  outline  in  their  1969  State  plans  a  tentative  program  for  pro- 
ject evaluation  and  the  measurement  of  overall  plan  performance.  ...  We  were  ad- 
vised by  the  LEAA  that  it  had  not  issued  guidelines  to  the  State  planning  agencies 
on  evaluation  methods  because  of  a  shortage  of  manpower.   Also  we  noted  that  on 
occasions,  information  or  guidance  has  been  requested  from  LEAA  on  monitoring  and 
evaluation  methods,  and  LEAA  has  been  unable  to  provide  the  assistance."   (The 
Block  Grant  Programs  of  the  Law  Enforcement  Assistance  Administration,  (Part  1), 
hearings,  op.  cit.,  pp.  143-4 
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the  more  imperative  since  State  and  local  governmental  units  have  been  given  greater 
responsibility  for  program  administration  and  accountability  under  revenue  sharing.  A 
few  agencies  have  encouraged  developments  in  this  area,  notably  Housing  and  Urban  De- 
velopment and  the  Office  of  Management  and  Budget.   42/  In  a  partial  effort  to  assist 
non-Federal  authorities  in  performing  required  audit  functions,  the  General  Accounting 
Office  prepared  a  report  in  June  1972,  titled  Standards  for  Audit  of  Governmental  Or- 
ganizations, Programs,  Activities,  and  Functions,  54  pages.   This  report  does  not  how- 
ever, provide  standards  for  program  evaluation.   The  GAO  is  now  preparing  a  program 
evaluation  guidebook,  expected  to  be  released  in  the  Spring  1975. 

5.   Proposals  for  Comprehensive  Evaluation  Plans.   A  number  of  agencies  conducting 
program  evaluation  research  have  begun  to  develop  coherent  evaluation  work  plans  to 
guide  both  internal  agency  evaluators  and  extramural  evaluators.   43/  As  will  be  noted 
below,  the  Office  of  Management  and  Budget  is  now  encouraging  activities  in  this  di- 
rection.  It  may  be  of  interest  to  the  Congress  to  inquire  into  whether  other  Federal 
agencies  are  taking  steps  to  provide  for  systematic  development  and  use  of  evaluation 
work  plans.  Who ley,  et.  al.  made  a  number  of  recommendations  treating  this  issue.  Some 
of  their  observations  are  reproduced  below: 


42/  See:   Ninth  Conference  on  Management  Analysis  in  State  and  Local  Government,  No- 
vember 2  and  3,  1972,  sponsored  by  Institute  of  Public  Service,  University  pf 
Connecticut.  Among  the  papers  were:   Program  Evaluation:  Pitfalls  and  Potential- 
ities, Dr.  Jack  Carlson,  OMB;  Case  History  Application  of  Program  Evaluation,  Keith 
Marvin,  General  Accounting  Office.   The  most  useful  study  in  this  respect  is  the 
HUD  funded  report:   Hatry,  Harry  P.  et.  al.   Practical  Program  Evaluation  for  State 
and  Local  Government  Officials.  Washington,  D.C.,  The  Urban  Institute,  September 
1972.   134  pp. 

43/  For  an  introduction  to  this  topic,  see  R.A.  Levine  and  A. P.  Williams,  Jr., 

Making  Evaluation  Effective:  A  Guide.  A  report  prepared  for  the  Department  of 
Health,  Education,  and  Welfare.   Santa  Monica,  The  RAND  Corporation,  May  1971,  40 
pp.   (R-788-HEW/CMU.)   Fred  Spaner  and  Charles  Winder,  National  Institutes  of 
Mental  Health.   The  Perspective  of  National  Program  Development  and  Administration: 
Program  Needs  and  Evaluation  Activities.  Unpublished,  February  5,  1971.  Presented 
at  American  Psychological  Association  Convention  September  1971.  Quotation  from 
Federal  Evaluation  Policy,  op.  cit.,  pp.  114-115. 
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At  the  beginning  of  each  fiscal  year,  each  agency  should  develop  or  update 
a  two  to  three-year  evaluation  work  plan  stating  which  studies  are  in  progress 
and  which  are  projected.  Approval  of  this  plan  should  be  required  at  the  be- 
ginning of  each  fiscal  year  as  a  condition  for  authority  to  spend  evaluation 
funds.   This  plan  should  be  prepared  by  the  agency-level  evaluation  staff  in 
cooperation  with  policy  makers,  budget  staff,  program  managers  and  operating- 
level  evaluation  staffs.   Evaluation  planning  is  a  continual  process:  when 
the  agency  appropriation  is  known,  the  evaluation  plan  should  be  reassessed 
in  light  of  firm  budget  figures. 

Since  useful  evaluation  studies  are  likely  to  be  costly,  agencies  should 
place  emphasis  on  feasible  studies  of  major  programs  where  the  value  of  the 
findings  would  outweigh  the  costs.   44/ 

6.   Proposals  for  a  "Preassessment  of  the  Evaluability  of  Programs."  As  a  variant 
of  this  theme,  researchers  at  the  Urban  Institute  suggest  that  the  conduct  and  use  of 
program  evaluation  will  be  enhanced  if  federal  agency  administrators  and  program  evalu- 
ation researchers  conduct  a  "preassessment  of  the  evaluability  of  a  public  program." 
The  objective  of  the  assessment  would  be  to  select  appropriate  research  strategies  for 
evaluation.   The  following  questions,  addressing  the  assumptions  of  the  program,  its 
data  base,  and  the  probable  use  of  the  evaluation  in  decisionmaking  processes,  suggest 
a  sequence  of  evaluation  steps  and  criteria  to  be  used  in  determining  the  "evaluability 
of  a  public  program"  and  priorities  among  different  types  of  evaluation  strategies: 

Are  the  problems,  intended  program  interventions,  anticipated  outcomes,  and 
the  expected  impact  sufficiently  well  defined  as  to  be  measurable? 

In  the  assumptions  linking  expenditure  to  implementation  or  intervention, 
intervention  to  the  outcome  anticipated,  and  immediate  outcome  to  the  ex- 
pected impact  on  the  problem,  is  the  logic  laid  out  clearly  enough  to  be 
tested? 

Is  there  anyone  clearly  in  charge  of  the  program?  Who?  What  are  the  con- 
straints on  his  ability  to  act?  What  range  of  actions  might  he  reasonably 
take  or  consider  as  a  result  of  various  possible  evaluation  findings  about 
the  measures  and  assumptions  discussed  above?  45/ 


44/   Federal  Evaluation  Policy,  op.  cit.,  pp.  114-5. 

45/  Horst,  Pamela,  Joe  N.  Nay,  John  W.  Scanlon,  and  Joseph  S.  Wholey.  Program  Manage- 
ment and  the  Federal  Evaluator.   Public  Administration  Review,  v.  4,  July/August 
1974:   307.   (Part  of  PAR/special  "Symposium  on  Program  Evaluation.")  (Footnote 
continued  on  p.  23.) 
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D.   The  Need  for  Better  Dissemination  of  Information  on  Program  Evaluations  Supported 
Under  Federal  Agency  Auspices 

Several  issues  are  subsumed  under  the  general  heading  of  the  need  for  improved 
program  evaluation  information  dissemination  systems.  Among  them  are:  the  need  for 
researchers  to  share  experiences  in  developing  methodologies  to  hasten  refinement  of 
program  evaluation  techniques;  to  eliminate  duplicate  funding  of  methodologies  and 
e\aluations  of  programs;  and  to  insure  distribution  of  program  evaluation  reports  to 
individuals  within  an  agency  and  to  individuals  outside  of  the  agency  whose  program 
responsibilities  intersect  with  the  program  being  evaluated. 

Howard  R.  Davis,  chief  of  the  Mental  Health  Services  Research  and  Development 

Branch  of  the  National  Institute  of  Mental  Health,  has  summed  up  these  concerns: 

Increasing  numbers  of  investigators  are  working  vigorously  on  methods  of  measure- 
ment and  analysis  in  evaluation;  but  one  researcher  hardly  knows  what  the  others 
are  doing.   Duplication  unavoidably  is  occurring.   Incompatible  counterparts  of 
evaluation  systems  are  needlessly  re-invented.   .  .  .  Better  methods  of  ex- 
changing information  [should]  be  developed  as  soon  as  possible.   Not  only  would 
each  person  producing  evaluation  knowledge  know  what  others  are  doing,  not  only 
would  those  needing  to  implement  evaluation  programs  have  a  valuable  perspective 
of  what  is  going  on  elsewhere,  but  the  concepts  of  evaluation.  .  .  would  receive 
continued  clarification.  .  .  .   ^6 / 

Perhaps  as  a  partial  response  to  this  need,  the  National  Institute  of  Mental  Health 
awarded  a  grant  to  the  Program  Evaluation  Project,  Minneapolis  Medical  Research  Founda- 
tion, to  establish,  on  an  experimental  basis,  Evaluation,  the  journal  in  which  this 
statement  is  found. 


The  National  Advisory  Council  on  Educational  Professions  Development  made  a 
related  recommendation  for  the  administration  of  each  specific  program  evaluation: 
"Evaluation  research  plans  can  fail  in  implementation  just  as  programs  can  Jail. 
Time  must  be  taken  to  'walk  through'  evaluation  research  designs  by  applying  them 
to  simulated  program  data,  operational  settings  and  decision  questions,  before 
applying  the  evaluation  research  design  to  the  actual  program.   Such  a  process 
could  create  much  more  understanding  of  various  models  of  evaluation  research  and 
alert  decision-makers  to  potential  problems  and  results  in  advance."   (Search  for 
success ,  op.  cit.,   24.) 

*jf     David,  Howard  R.   Insights:   A  Solution  for  Crisis?  Evaluation,  v.  I,  no.  1,  Fall 
1972:   3-4. 
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1.  Investigation  by  the  General  Accounting  Office.   A  number  of  recent  reports 
indicate  that  far  more  systematic  methods  of  exchanging  information  on  program  evalua- 
tion may  be  required.   For  example,  the  General  Accounting  Office  published  a  report 
entitled  "Program  Evaluation:   Legislative  Language  and  a  User's  Guide  to  Selected 
Sources,"  in  June  1973.   This  report  was  prepared  to  assist  GAO  researchers  obtain  pub- 
lished information  and  program  evaluations  to  which  they  could  refer  in  performing  their 
auditing  and  evaluation  functions  for  the  Congress.   The  author  describes  and  assesses 
30  on-going  research  inventory  and  published  information  sources  established  by  Federal 
agencies.   These,  among  others,  include  the  National  Technical  Information  Service 
(NTIS) ;  the  Science  Information  Exchange  (SIE);  ERIC,  a  research  report  inventory  for 
federally  funded  education  research;  and  MEDLARS  and  NCHSRD  (Inventories  of  research  in 
progress  or  published  related  to  medical  and  mental  health  services,  respectively). 

The  report  indicates  the  following  inadequacies: 

Despite  [the]  rapid  growth  in  evaluation  studies,  information  as  to  who  is 
conducting  which  studies  with  what  findings  is  not  readily  available.  No  single- 
source  document  or  retrieval  system  identifies  planned,  ongoing,  and  completed 
evaluations.   No  single  source  describes  the  findings  and  conclusions  of  completed 
evaluations.   In  fact,  no  single  functional  area  (such  as  health,  education,  or 
training)  has  such  complete  coverage  of  evaluation  studies.   Since  program  evalua- 
tion provides  perhaps  the  most  comprehensive,  objective,  and  quantitative  sources 
of  Information  on  governmental  program  results,  it  is  Important  to  the  government 
and  to  the  public  that  these  results  be  identified  and  made  available.  Most  non- 
defense  evaluations  are  public  information  and  need  to  be  listed  and  described. 

A  number  of  computer  retrieval  systems  and  reference  documents  listing  a 
large  number  of  studies,  including  evaluation  studies,  are  available.   However, 
knowledge  is  generally  lacking  as  to  which  information  sources  exist,  which  sources 
contain  evaluation  studies,  how  comprehensively  evaluation  studies  are  described, 
and  how  to  obtain  information  from  such  sources.   This  chaotic  situation  makes  it 
a  long-term  task  to  develop,  maintain,  and  distribute  a  comprehensive  list  and 
description  of  planned,  ongoing,  and  completed  evaluation  studies  in  all  functional 
areas .  Ujl 

2.  NIMH/UCLA  Project  on  Data  Bank  of  Program  Evaluation  (DOPE).   It  should  be 
noted  that  the  Health  Services  and  Mental  Health  Development  Branch  of  the  National 


47/  Program  Evaluation:   Legislative  Language  and  a  User's  Guide  to  Selected  Sources, 
op.  clt.,  p.sl. 
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Institute  of  Mental  Health  is  funding  a  joint  project  with  UCLA  to  establish  an  easily 
accessible  computerized  information  system  on  program  evaluation,  called  the  Databank 
of  program  Evaluations  (DOPE).   The  system,  expected  to  be  in  operation  in  early  1975, 
will  enable  all  researchers  who  have  access  to  the  ARPA  computer  network  (of  the  Ad- 
vanced Research  Projects  Agency)  to  obtain  an  abstract  of  the  following  information  for 
each  program  evaluation  report  and  secondary  source  document  included  in  the  system: 

-citation^ 

-status  of  the  program  success, 

-condition  treated 

-age,  Be*,  race  and  income  characteristics  of  the  sample  population, 

-sample  size, 

-site, 

-treatment  method, 

-characteristics  of  the  study  design, 

-what  is  measured, 

-what  measures  are  used, 

-how  data  Is  collected,  and 

-conclusions. 

Evaluation  reports  will  encompass  the  following  topics!   alcohol  abuse,  divorce,  juvenile 
delinquency,  neuroses,  sexual  deviancy,  suicide,  welfare,  health  problems  with  mental 
health  outcomes,  criminality,  drug  ahu.se,  mental  retardation,  psychoses,  socioculfcural 
problems,  unemployment,  and  mental  problems  of  the  aged.   Three  types  of  search  are 
being  used  to  Identify  reports  of  evaluations:   (1)  journal  articlesj  (2)  published  and 
unpublished  reports  recommended  by  experts  in  each  subject  field,  and  (3)  materials 
from  other  computerized  data  banks,  such  as  Medlars,  Psychological  Abstracts,  and  Socio- 
logical Abstracts.  48/ 


^8/  Wilner,  Daniel  M.,  Robert  W.  Hetherington,  Ellen  B.  Goid,  Daniel  H.  Ershoff,  and 
Cedric  F.  Garagliano.  Databiink  of  frogram  Evaluations.  Evaluation,  V.  I,  no.  3, 
1973:   3*6.   (The  project  is  supported  under  grant  IR  12-MH-2276.) 
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PART  III.   IMPROVED  COORDINATION  OF 

PROGRAM  EVALCATION  IN  THE  EXECUTIVE 

AND  LEGISLATIVE  BRANCHES 

A; Issues  Surrounding  the  Need  to  Incorporate  Program  Evaluation  Findings  Into 

Executive  Branch  Budget  Accesses 

Two  topics  are  included  under  this  heading  of  improved  coordination  of  program 
evaluation  in  the  executive  branch.   The  first  relates  to  ways  to  further  incorporate 
program  evaluation  findings  into  executive  branch  budget  processes.   The  second  deals 
with  imprcved  coordination  of  the  administration  and  use  of  program  evaluation  in  the 
executive  branch. 

The  c?r.ly  government -wide  requirements  fcr  agencies  to  incorporate  program 
evaluation  data  into  an  jual  agency  budget  proposals  are  found  in  Circular  A-ll,  issued 
by  the  Office  of  Management  and  Budget.   The  term  "program  evaluation"  is  used  very 
loosely  i  R   this  circular,  since  costs  anc.  benefits  or  program  accomplishments  or  ont- 
?utF.  are  to  be  -stated  directly  in  dollar  terms  or  in  number  of  persons  affected, 
rather  than  in  more  hard  to  measure  terms  of  real,  but  mostly  intangible,  program 
accomplishments.   For  instance,  data  to  be  submitted  for  Federal  Income  Security 
Programs  consist  of  three  types:  benefit  outlays,  number  of  beneficiaries  and 
average  monthly  benefits,  number  of  individuals  benefited  by  the  outlays  and  number 
of  primary  beneficiaries.   These  data  are  required  of  separate  target  groups  cate- 
gorized as  aged  or  retired;  disabled;  mothers;  surrvivinf,  spouse;  other  survivors; 
unemployed  and  others.  (Sec.  49.)  These  data  requirements  differ  considerably  from 
requirements  for  most  agency  program  evaluations,  which  are  designed  to  determine 
whether  a  program  should  be  modified,  terminated,  or  continued.   Evaluation  require- 
ments under  Circular  A-ll  also  do  not  yield  information  to  determine  which  among 
alternative  social  programs  best  serves  a  particular  target  group  or  meets  an  accepted 
national  need.  These  requirements  have  also  been  videly  criticized  as  inappropriate 
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to  the  budgeting  process  in  both  executive  and  legislative  branches  since  almost  no 
attempt  is  made  by  the  Office  of  Management  and  Budget  to  integrate  the  results  of 
program  evaluations  and  program  evaluation  information  to  the  annual  budget  process. 49/ 
Criticism  has  been  heard  also  that  what  information  the  executive  branch  does  collect 
is  withheld  from  the  Congress  all  too  frequently  on  the  grounds  of  executive  privilege. 
Commenting  on  this  point  in  1970,  (when  Office  of  Management  and  Budget  was  called 
the  Bureau  of  the  Budget),  the  Joint  Economic  Committee  said: 

Program  Memoranda,  containing  quantitative  evaluation  of  the  benefits  and 
the  costs  of  individual  programs  as  well  as  other  evaluative  information,  have 
been  retained  ...  on  grounds  of  executive  privilege.   Special  Analytic  Studies, 
presenting  the  results  of  detailed  study  of  the  efficiency  and  equity  consequences 
of  alternative  means  of  obtaining  objectives,  are  from  time  to  time  released,  but 
only  after  the  executive  has  made  a  policy  decision  and  then  only  in  a  form 
designed  not  to  upset  the  decision.   Program  and  Financial  Plans,  with  program 
budget  projections,  are  held  within  the  executive  on  grounds  of  lack  of  data 
and  the  unreliability  of  estimates.   While  the  list  of  issues  on  which  the 
Bureau  of  the  Budget  is  requesting  analysis  in  the  planning  and  budgeting  cycle 
has  been  released  to  the  Congress  through  the  subcommittee,  the  results  of  the 
analyses  of  these  issues  have  been  withheld  . . . 

...  The  lack  of  congressional  access  to  much  of  this  information  and  analysis 
seriously  hinders  legislative  efforts  to  scrutinize  and  control  the  budget.   The 
widespread  use  of  executive  privilege  to  retain  the  results  of  program  evaluation 
and  analysis  is  inimical  to  the  development  of  an  open  and  responsive  political 
system  in  which  public  policy  is  made  by  elected  political  leaders  rather  than 
by  an  anonymous  and  powerful  bureaucracy .5°/ 

And  although  0MB  asks  agencies  to  describe  generally  how  programs  meet  national 

objectives,  termination  of  requirements  for  agencies  to  adhere  to  the  Planning, 

Programming,  Budgeting  system,  in  effect,  removed  requirements  for  agencies  to  analyze 


^ I     Allen  Schick,  "An  Analysis  of  Proposals  to  Improve  Congressional  Control  of 
Spending."  In  U.S.  Congress.   Joint  Economic  Committee.   Subcommittee  on 
Budgeting,  Management  and  Expenditures.   Improving  Congressional  Control  Over 
the  Budget:   ACompendium  of  Materials.  March  27,  1973.   Washington,  U.S. 
Govt.  Print.  Off.,  1973.   pp.  223-4.   (Committee  Print.) 

_/  U.S.  Congress.   Joint  Economic  Committee.   Subcommittee  on  Economy  in  Government. 
Economic  Analysis  and  the  Efficiency  of  Government.   Report.   February  9,  1970. 
Washington,  D.C.,  U.S.   Govt.  Print.  Off.,  1970.  p.  9.   (Joint  Committee  Print.) 
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program  accomplishments  and  deficiencies  In  relation  to  program  planning.  51/ 


51/  General  Information  on  removal  of  requirements  for  PPB,  stemming  from  admini- 
strative problems  and  difficulties  in  obtaining  required  information,  are 
reviewed  in:   Schick,  Allen.   A  Death  In  the  Bureaucracy:   The  Demise  of  Federal 
PPB  Public  Administration  Review,  vol.  33,  March/April,  1973:   146-156. 
Specific  details  on  requirements  for  submitting  evaluation  data  in  budget 
proposals  and  on  the  evolution  of  PPB  requirements  were  supplied  by  Mr.  Joel 
Anderson,  Office  of  Management  and  Budget: 

According  to  Mr.  Anderson  the  most  important  0MB  directive  on  program  evaluation 
is  Bulletin  No.  68-9,  issued  April  12,  1968,  supplemented  specifically  by 
Bulletin  No.  68-9,  supplement  No.  1,  July  17,  1969,  and  supplemented  generally 
by  Circular  No.  A- 11,  usually  revised  and  re-issued  each  year.   Bulletin  No. 
68-9  and  supplement  No.  1  treat  the  Planning-Programming-Budgeting  System. 
Circulars  A-ll  treat  preparation  and  submission  of  annual  budget  estimates. 

These  0MB  directives  treat  program  evaluation  only  very  generally  within  the 
context  of  "programming"  as  part  budgeting/appropriations  submissions  to  the 
Congress.   Effectiveness  is  treated  in  this  sense  from  an  economic  approach. 
The  "programs"  which  are  to  be  evaluated  for  the  purposes  of  preparing  data 
required  by  these  directives  are  general  budget  categories,  not  specific  oper- 
ational agency  programs. 

President  Richard  M.  Nixon  issued  a  Memorandum  to  the  heads  of  Federal  departments 
and  agencies,  May  25,  1970,  In  which  he  outlined  the  importance  of  evaluating 
effectiveness  and  using  program  evaluation  to  underpin  agency  planning  and 
budgeting  activities.   He  outlined  general  criteria  which  the  required  program 
evaluations  were  to  meet.   0MB  circular  A-ll,  1971  (With  transmittal  Memorandum 
No.  38) ,  reflects  the  need  for  agencies  to  meet  these  general  criteria  in 
preparing  budget  estimates.   However,  the  criteria  outlined  in  the  Circular  are 
discretionary  and  diverge  somewhat  from  the  specific  requirements  under  the 
rescinded  PPB  guidelines.   ("Analysis  of  Changes  in  Office  of  Management  and 
Budget  Circular  No.  A-ll,  Revised  June  1971,  Comparison  of  Old  Sec.  No. 
24.1-24.6  to  24.1-24.5,  p.  3.) 

Program  evaluation  requirements  in  the  current  circular,  A-ll,  Revised, 
Transmittal  Memorandum  No.  39,  June  12,  1972,  are  similar  to  the  generalized 
requirements  of  the  1971  circular.   These  requirements  are  outlined  in 
paragraphs  24.2-24.3  (Analysis  and  Justification  of  Programs  and  36.1-2 

Narrative  Statements  on  Program  Performance.)   The  material  requested  to 
fulfill  requirements  for  "Analysis  and  Justification  of  Programs"  supports 
the  first  part  of  the  budget  request;  material  required  to  meet  "Narrative 
Statements  on  Program  and  Performance"  in  the  circular  Is  used  for  detailed 
budget  appendix.   Mention  is  made  of  a  number  of  techniques  for  conducting 
evaluations,  and  for  relating  program  objectives  and  inputs  to  program  benefits 
and  social  benefits.   However,  the  Circular  seems  to  leave  It  up  to  the  discretion 
of  the  agency  director  to  use  techniques  which  are  most  appropriate  in  evaluating 
the  effectiveness  of  Its  programs. 

Specific  "evaluation"  requirements  for  programs  falling  under  separate  functional 
areas,  such  as  manpower,  health,  and  education,  are  also  detailed  in  Circular  A-ll. 
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B.  Proposals  to  Give  the  Office  of  Management  and  Budget  Program  Evaluation 
Responsibilities 

A  number  of  proposals  have  been  made  to  give  the  Office  of  Management  and 

Budget  explicit  coordinating  and  governing  responsibilities  for  program  evaluations 

conducted  by  agencies  of  the  Government.   Joseph  S.  Wholey,  et.  al.,  examined  this 

issue  in  a  1970  report  which  analyzed  the  evaluation  activities  of  four  Federal 

agencies.   They  reported:   "The  executive  branch  has  no  system  for  Identifying 

major  questions  to  be  answered  through  evaluation,  for  ensuring  that  high  priority 

studies  were  carried  out,  or  for  ensuring  that  significant  evaluation  findings 

reached  appropriate  decisionmakers."  They  recommended  that:   the  Bureau  of  the 

Budget  (now  called  0MB)  (a)  require  agencies  to  develop  two-to-three  year  evaluation 

plans  and  to  use  evaluation  information  in  budgetmaking  and  legislative  review; 

(b)  evaluate  the  quality  of  evaluation  techniques  used  by  different  agencies; 

(c)  conduct  cross-agency  evaluations  "to  compare  the  effectiveness  of  related 
Federal  programs  in  achieving  common  objectives;"  and  (d)  enlarge  budget  allocations 
for  evaluation  staff  and  resources. 52/ 

C.  The  Proposal  to  Establish  a  Federal  Program  Evaluation  Office  in  Either  the 
Office  of  Management  and  Budget  or  in  the  Executive  Office  of  the  President 

Laurence  E.  Lynn,  Jr.,  who  served  as  Assistant  Secretary  for  Planning  and 

Evaluation  in  the  Department  of  Health,  Education  and  Welfare,  between  1971  and 

early  1973,  made  the  most  recent  proposal  in  this  respect.   In  order  to  overcome 


^2/  Federal  Evaluation  Policy,  op.  cit.,  24,  58-61.   Similar  recommendations  have 
been  made  by  the  Committee  for  Economic  Development  in  its  report  Improving 
Federal  Program  Evaluation,  op.  cit.,  59  and  by  the  President's  Advisory 
Council  on  Executive  Organization.  Memorandum  for  the  President  of  the  United 
States  on  Establishment  of  a  Department  of  Natural  Resources  [and]  Organi- 
zation for  Social  and  Economic  Programs  (including  background  memoranda  and 
attachments).   February  5,  1971,  Appendix  5. 
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CRS  -  30 

Che  many  inadequacies  of  current  program  evaluation  practices,  he  proposed  the  creation 
of  a  Federal  Program  Evaluation  Office  in  either  the  office  of  Management  and  Budget  or 
in  the  Executive  Office  of  the  President  to  "[supervise]  the  evaluation  activities  of 
federal  program  agencies,  [to  conduct]  a  program  of  research  on  evaluation  methods,  and 
[to  produce]  an  annual  evaluation  report. "53/  Specifically  the  proposed  Federal  Pro- 
gram Evaluation  Office  (FPEO)  would: 

. . .  determine  how  to  apportion  the  funds  among  the  FPEO  and  the  agencies  in 
accordance  with  a  comprehensive  plan  .... 

...  be  directly  responsible  for,  and  retain  full  authority  over,  all  program 
impact  evaluations.  Adequate  qualified  staff  would  be  maintained  to  do  the 
necessary  planning,  write  requests  for  proposals,  review  proposals,  and  most 
important,  closely  monitor  the  work  of  the  evaluation  contractor  or  grantee. 

...  be  responsible  for  overseeing  the  development  of  plans  for  program  strategy 
evaluations  and  project  evaluations  .... 

. . .  prepare  the  annual  evaluation  report  summarizing  evaluation  findings  and 
work  in  progress  .... 

. . .  operate  a  system  for  rating  the  performance  of  evaluation  contractors  and 
grantees  and  for  making  the  ratings  available  to  evaluation  officials. 

—  [develop  a  priority  framework]  to  guide  the  allocation  of  evaluation 
resources  ...  in  recognition  of  resource  limitations,  the  potential  value  of 
possible  valuations,  and  the  availability  of  evaluation  capacity. 54/ 

D.   Current  OMB  Initiatives  in  Program  Evaluation 

As  noted  above,  the  OMB  has  stepped  up  its  evaluation  activities  in  order  to 
improve  the  management  and  use  of  evaluation  research.   In  a  recent  article  two  OMB 
officials,  who  were  then  responsible  for  directing  the  agency's  program  evaluation 
activities,  outlined  the  specific  objectives  of  the  present  OMB  effort.   These 
objectives,  directed  at  each  Federal  agency,  are: 


53/  Lynn,  Laurence  E.  Jr.  A  Federal  Evaluation  Office?  Evaluation,  v.  I,  no.  2, 
1973:   29, 62. 

54/  Ibid.,  pp.  62,96. 
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(a)  development  of  built-in  evaluation  capability  from  the  inception  of  each 
new  Federal  program;  (b)  adequate  staff  to  assure  quality  of  output;  (c) 
sufficient  planning  to  focus  attention  on  the  high  pay-off  areas;  (d)  follow-up 
procedures  to  assure  that  results  are  utilized  (e)  organizational  alignments 
to  assure  full  integration  with  the  policy  decision  process  and  related  and 
supportive  activities;  and  (f)  a  proper  balancing  of  long-term  and  short-term 
efforts  (in-house  and  external)  to  meet  both  quality  and  timeliness  needs. 55/ 

Briefly,  OMB's  evaluation  activities  consist  of  the  following:   (1)   establishment 
of  a  Division  of  Evaluation  and  Program  Implementation  under  the  Associate  Director 
for  Management  and  Operations;  the  Office  is  directed  by  Mr.  Clifford  W.  Graves; 
(2)  establishment  "on  a  very  informal  basis  of  a  panel  of  senior  Federal  executives 
to  serve  as  a  forum  for  the  mutual  exchange  of  ideas  and  developments  within  the 
Federal  evaluation  community;"   (3)  initiation  of  a  "...  comprehensive  survey  of 
Federal  evaluation  programs  that  will  inventory  and  catalogue  ...  such  activities;" 
(4)  development  of  a  "cross-agency  budget  review  to  assess  the  adequacy  of  funding 
levels  for  evaluation  programs  in  a  number  of  Federal  agencies;"  (5)  encouragement 
of  all  agencies  to  prepare  "formal  plans  on  a  periodic  basis  to  direct  and  guide 
individual  revaluation]  studies  to  high  priority  topics;" (6)  review  of  legislative 
initiatives  "to  assure  that  adequate  provision  is  made  for  evaluation"  and  for 
establishment  of  an  appropriate  evaluation  capability  for  the  program;  (7)  incor- 
poration of  evaluation  priorities  into  the  management  by  objectives  effort;  and 
(8)  monitoring,  on  a  selective  basis,  of  evaluation  studies  of  particular  interest 
to  OMB  and  the  White  House. 56/ 

As  a  first  step  in  carrying  out  these  responsibilities,  the  Evaluation  and 
Program  Implementation  Division  of  OMB  conducted  a  survey  of  federal  agency 


55/  Federal  Program  Evaluation  From  the  OMB  Perspective,  op.  cit.,  p.  317;  each 
of  these  objectives  is  fully  described  on  pp.  314-317. 

56/  Ibid.,  pp.  312-314. 
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evaluation  activities  which  provides  information  on  the  following  points: 

-  persons(s)  in  the  agency  responsible  for  performing  different  types  of 
evaluations ; 

-  types  of  activities  undertaken; 

-  magnitude  of  resources  available  for  evaluation; 

-  characterization  of  relationships  between  the  production  of  evaluation 
information  and  decision  processes  such  as  management  by  objectives,  budget 
preparation,  and  statistical  planning;  and 

-  persons  who  can  provide  additional  information  on  evaluations  conducted. 

A  draft  version  of  the  report  was  released  in  June.   It  is  expected  that  the 

materials  will  be  updated  and  disseminated  more  widely  after  additional  agencies 

provide  responses  to  the  OMB  questionnaire. 57/ 

E.   Issues  Relating  to  the  Improvement  of  Legislative  Branch  Program  Evaluation 
Capabilities 

The  absence  of  a  congressional  program  evaluation  capability  has  repeatedly 
been  cited  as  a  major  impediment  to  incorporating  program  evaluation  reports  into 
the  legislative  process.   Two  frequently  mentioned  themes  are:   (1)  that  the  executive 
branch  both  conducts  program  evaluations  and  assesses  the  results  of  program  evalua- 
tions without  presenting  reports  to  the  Congress  or  to  an  objective  critic, 58/ 


57/   Survey  of  Federal  Evaluation  Activities.   By  Evaluation  and  Program  Implemen- 
tation Staff,  Office  of  Management  and  Budget,  June  1974,  draft,  loose  leaf, 
various  pagings. 

58/  On  this  point  Schick  reports:   "...  Evaluation  comes  too  late,  after  the 

program  has  garnered  strong  client  support  and  is  built  into  the  budget  base. 
The  classic  case  is  summer  Headstart  which  was  serving  hundreds  of  thousands 
of  preschoolers  when  adverse  findings  were  released.   Popular  programs  are 
hard  to  terminate,  no  matter  what  judgments  flow  from  the  evaluations."  ("The 
Pilot  Testing  of  New  Programs:  An  Analysis  of  Title  IV,  S.  3984,"  In: 
Improving  Congressional  Control  of  the  Budget,  op.  cit,   p.  278) 
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and  (2)  the  Congress  does  not  have  necessary  staff  expertise  to  analyze  such  reports 

even  if  they  were  provided.   Murray  L.  Weidenbaum,  et.  al.  summarized  these  problems: 

Evaluative  research  is  now  a  virtual  monopoly  of  the  executive  branch,  to 
the  extent  that  it  is  used  at  all.   If  Congress  is  ever  to  oversee  executive 
administration  effectively,  then  it  needs  professional  assistance  beyond  that 
provided  by  existing  committee  staffs.   As  imperfect  as  evaluation  techniques 
are,  at  the  least  they  would  be  a  useful  supplement  to  existing  rules -of -thumb 
and  other  subjective  approaches. 59/ 

In  this  same  respect  the  Joint  Economic  Committee  reported: 

The  relationship  of  the  executive  and  legislative  branches  pertaining  to 
the  use  of  executive  analysis  and  evaluation  documents  is  a  serious  problem. 
More  serious,  however,  is  the  failure  of  the  Congress  to  provide  itself  with 
an  analytical  capability.   Currently,  the  Congress  has  neither  an  adequate 
capability  to  interpret  or  evaluate  studies  done  by  the  executive  and  those 
outside  of  Government  nor  the  necessary  capacity  to  undertake  policy  analysis 
of  its  own. 60/ 

Evaluation  of  public  welfare  programs  would  be  enhanced,  the  Joint  Economic  Committee 
reported  in  November  1972,  by  creating  an  evaluation  agency  independent  of  the  ad- 
ministering agency  and  reporting  directly  to  the  Congress: 

.  .  .  Enabling  legislation  usually  assigns  the  task  of  evaluation  to  the  program 
administrators.   Separation  of  powers  is  a  well-accepted  and  venerable  principle, 
and  its  application  is  as  appropriate  here  as  elsewhere.   Administrators  are 
understandably  anxious  to  depict  their  programs  as  successful,  and  evaluations 
conducted  by  them  (no  matter  how  conscientious  they  may  be)  cannot  escape  being 
suspected  of  bias.   An  independent  agency  accountable  to  Congress,  should  be 
responsible  for  evaluation. 61/ 


59/  Weidenbaum,  Murray  L.,  Dan  Lark ins  and  Philip  N.  Marcus.   Matching  Needs  and 
Resources:   Reforming  the  Federal  Budget.  Washington,  D.C.,  American  Enter- 
prise Institute  for  Public  Policy  Research  [1973]  p.  11. 

60/  U.S.  Congress.  Joint  Economic  Committee.   Studies  in  Public  Welfare,  Paper  No. 
3,  The  Effectiveness  of  Manpower  Training  Programs:   A  Review  of  Research  on 
the  Impact  of  the  Poor.   A  staff  study  prepared  for  the  use  of  the  Subcomm.  on 
Fiscal  Policy.   November  20,  1972.  Washington,  U.S.  Govt.  Print.  Off.,  1972. 
p.  13.   (Joint  Committee  Print.)  The  other  studies  in  the  series  are:  Paper  no. 
1,  Public  Income  Transfer  Programs:   The  Incidence  of  Multiple  Benefits  and 
the  Issues  Raised  by  Their  Receipt,  April  10,  1972;  Paper  No.  2,  Handbook  of 
Public  Income  Transfer  Program:   How  They  Tax  the  Poor,  December  22,  1972; 
Paper  No.  5,  Part  1.   Issues  in  Welfare  Administration:   Welfare An  Admin- 
istrative Nightmare,  December  31,  1972. 

£1/  The  Effectiveness  of  Manpower  Training  Programs:   A  Review  of  Research  on  the 
Impact  of  the  Poor,  op.  cit.,  p.  15. 
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Within  the  last  year  or  two  proposals  were  made  to  provide  the  Congress  with 
a  program  evaluation  capability  independent  of  the  executive  branch.  Most  of  these 
responded  to  alternative  ways  of  meeting  some  of  the  recommendations  of  the  Joint 
Study  Committee  on  Budget  Control  (■"_'     One  of  the  committee's  major  observations 
was  that  legislative  determination  of  priorities  among  programs  and  of  spending  levels 
for  these  programs  would  be  strengthened  if  the  Congress  coordinated  more  closely  the 
appropriations  process  and  the  authorization  process  (which  now  fragments  legislative 
review  of  the  evaluation  of  the  effectiveness  of  particular  programs,  evaluation  of 
the  merits  of  one  program  against  another,  and  program  funding).^/ 

During  the  93rd  Congress,  both  Houses  directed  attention  to  improving  the  process 
of  determining  and  evaluating  budget  priorities  to  serve  national  needs  and  to 
strengthening  the  review  of  the  budget  process  and  evaluation  of  Federal  programs. 
The  Congressional  Budget  and  Impoundment  Control  Act,  P.L.   93-344,  July  23,  1974, 
includes  two  sections  which  expand  legislative  program  evaluation  capabilities.  Under 
section  701,  committees  are  given  the  authority  to  conduct  evaluations  themselves, 
using  pilot  tests  and  cost/benefit  analyses,  if  warranted,  or  to  require  agencies 
to  evaluate  programs  and  report  the  results  to  committees.   Section  702  expands  the 
review  and  evaluation  functions  of  the  General  Accounting  Office.  Under  this  section 
the  GAO  is  charged  with  responsibilitv  for  assisting  committees  in  evaluating  govern- 
ment programs,  including  the  development  of  statements  of  legislative  objectives, 


62/  The  committee  was  created  in  October  1972,  pursuant  to  P.L.  92-599,  to  recommend 
procedures  for  improved  congressional  control  over  the  outlay  and  receipt  totals 
in  the  budget  and  to  help  establish  spending  priorities  in  the  budget.   The 
committee  presented  its  interim  report  to  the  Congress  on  February  6,  1973, 
"Improved  Congressional  Control  Over  Budgetary  Outlay  and  Receipt  Totals." 

63/  See  section  II.   "Tentative  Recommendations.   Introductory  paragraph."   (In 

Interim  Report  by  the  Joint  Study  Committee  on  Budget  Control.   Remarks  of  Hon. 
Mr.  Ullman  on  the  floor  of  the  House.   Congressional  Record,  February  7,  1973: 
H.   843.) 
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methods  for  review  and  evaluation  of  programs,  and  the  analysis  of  program  results. 
The  Act  also  authorizes  the  establishment  of  an  Office  of  Program  Review  and  Evaluation 
in  the  General  Accounting  Office. 64/ 

Some  of  the  other  bills  considered  during  the  93rd  Congress  included  provisions 
which  would  have  made  pilot  testing  mandatory  or  would  have  required  OMB  to  inventory 
or  to  use  program  evaluation  information  in  preparing  budget  submittals.   (S.  40,  Mr. 
Brock;  S.  1030,  Mr.  Humphrey:  S.  758,  Mr.  Beall;  and  S.  834,  Mr.  Muskie.   S.  758,  in 
particular,  would  have  created  a  Federal  program  evaluation  digest.) 

Related  to  this  was  a  House  proposal,  H.  Con.  Res.  21,  introduced  by  Mr.  Conte 

and  referred  to  the  Committee  on  Rules,  which  would  establish  a  "Joint  Select  Committee 

on  Government  Program  Analysis  and  Evaluation."  The  Committee  would  be  authorized  to 

evaluate  alternative  procedures  and  to  evaluate  "the  efficiency  and  effectiveness  of 

Federal  programs  and  activities  by  objective,  scientific,  and  empirical  analysis...." 

Among  the  alternative  proposals  the  Committee  would  examine  are: 

-"the  establishment  of  an  ad  hoc,  independent,  bipartisan  commission  to 
study  and  appraise  Federal  programs  and  activities. . .of  the  executive  branch...;" 

-"an  expansion  of  the  role  and  function  of  existing  agencies  and  authorities 
(such  as  the  General  Accounting  Office)  within  or  outside  of  Congress  to  perform 
such  analyses  and  evaluations ; " 

-"improved  staffing  of  standing  committees. . .through  the  addition  of  review 
specialists  and  persons  scientifically  trained  and  experienced  in  the  area  of 
program  audit  and  evaluations;"  and 

-"the  establishment  of  a  central  staff  or  office  as  an  integral  part  of  the 
Congress  charged  with  performing  program  audits,  evaluations,  and  analyses  under 
the  supervision  of  a  legislative  auditor  and  under  the  general  supervisory 
control  of  a  joint  committee  of  the  Congress. . ."65/ 


£*_/  Title  VTI  -  Program  Review  and  Evaluation,  Conference  Report,  H.R.  7130, 

Congressional  Budget  and  Impoundment  Control  Act,  passed  Senate  June  21,  1974. 
P.L.  93-344. 

eJ_l     Sec.  1  (b),  H.  Con.  Res.  21,  January  3,  1973. 
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F.   Issues  Surrounding  the  Role  of  the  General  Accounting  Office  In  Providing  the 

Congress  with  Improved  Program  Evaluation  Capability 

The  General  Accounting  Office's  role  in  providing  the  Congress  with  program 
evaluations  independent  of  the  executive  branch  has  received  considerable  attention 
with  the  last  two  years.   Several  issues  are  Involved:   (1)  Are  the  agency's  auditing, 
accounting,  and  program  evaluation  related  activities  sufficient  to  satisfy  legisla- 
tive requirements  for  program  evaluation  and  evaluation  of  the  utility  of  different 
program  intervention  strategies  in  meeting  legislative  goals?   (2)  Does  the  agency 
have  sufficient  authority,  staffing  and  appropriate  administrative  arrangements  to 
conduct  program  evaluations?   (3)  How  can  the  GAO  and  the  Congress  work  more  closely 
in  developing  program  objectives  and  legislative  language  which  meets  requirements 
for  the  conduct  of  program  evaluation? 

1.   Basis  of  GAO's  Statutory  Authority  for  Program  Evaluation.   Comptroller 
General  Elmer  B.  Staats,  noted  the  basis  of  the  agency's  statutory  authority  for  the 
conduct  of  program  evaluation  in  a  recent  article:   "While  the  GAO  has  always  construed 
the  Budget  and  Accounting  Act  and  the  Legislative  Reorganization  Act  of  1946  to  include 
this  authority,  the  Legislative  Reorganization  Act  of  1970  made  it  quite  explicit. 
This  act,  in  brief,  directed  that  GAO,  either  on  its  own  initiative  or  at  the  request 
of  committees  of  the  Congress,  make  studies  of  the  costs  and  benefits  of  existing 
programs. "66/  He  continued  that  the  agency  has  actively  carried  out  its  responsi- 
bilities under  these  acts: 

For  the  past  5  years,  GAO  has  given  high  priority  to  the  evaluation  of  Federal 
programs  to  the  point  where  approximately  30  percent  of  Its  professional  staff 
of  3,250  is  now  engaged  in  evaluations  and  studies  with  this  objective. 67/ 

2.   Evolution  of  GAO's  Program  Evaluation  Activities.   Issues  have  been  raised 
regarding  the  nature  of  GAO's  evaluation  activities.   For  instance,  Senator  Roth, 


66  /   Staats,  Elmer  B.   Improving  Congressional  Control  Over  the  Federal  Budget.   The 
GAO  Review,  Summer  1973:   11. 

6?/   Improving  Congressional  Control  Over  the  Federal  Budget,  op.  cit.:  p. 11. 
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consenting  upon   responses   received   in  his   staff   survey  of  program  evaluation  practices 

noted: 

A  rather  small  portion  of  the  executive  departments,  and  an  even  smaller 
portion  of  the  independent  agencies  indicated  in  response  to  our  inquiry  that 
GAO  was  actively  or  regularly  Involved  in  evaluating  the  substantive  accom- 
plishments of  their  programs.   They  also  stated  that  the  Comptroller  General's 
interest  in  their  programs  was  quite  often  of  a  fiscal-procedural  nature.68/ 

Continuing,  he  recommended  that  the  GAO's  program  evaluation  responsibilities  and 

those  of  the  other  congressional  information  support  agencies,  including  the  Library 

of  Congress  and  congressional  committee  staffs,  should  be  expanded  to  include  more 

program  evaluation  impact  analysis: 

It  is  clear  that  GAO  has  plenty  of  work  to  do  and  does  much  of  it  effectively. 
However,  Congress  needs  to  have  more  independent  evaluation  of  the  impact  of 
Federal  governmental  activity by  GAO,  the  Library  of  Congress,  its  own  com- 
mittees, or  perhaps  by  some  other  body.   The  Legislative  Reorganization  Act  of 
1970  clearly  assigns  to  the  Comptroller  General  and  the  Library  of  Congress 
additional  responsibility  to  perform  substantive  evaluations. 69/ 

In  answering  questions  posed  by  critics  of  GAO's  program  evaluation  efforts,  the 

Comptroller  General  admits  that  "GAO  still  has  much  to  learn,"  but  that  "overall  good 

progress  is  being  made."  "Evaluation  of  government  program  results,"  he  continues, 

"is  an  art  about  which  all  of  us  have  much  to  learn."  The  agency  faces  two  major 

difficulties  in  conducting  evaluations,  "particularly  in  the  social  action  areas:" 

Not  the  least  of  these  are  (1)  the  lack  of  clearly,  specifically  stated 
program  goals  and  objectives  and  (2)  the  lack  of  reliable  data  on  performance 
and  effects  or  results  of  program  operations. 70/ 

In  another  article  Assistant  Comptroller  General  Ellsworth  H.  R.  Morse  reported 

that  the  agency  is  attempting  to  improve  training  and  administrative  procedures  to 

assist  GAO  in  meeting  its  program  performance  and  evaluation  responsibilities: 


68/  Public  Program  Analysis  and  Evaluation  for  the  Purposes  of  the  Executive  and  the 
Congress,  op.  cit.:  S9026. 

69/     Idem. 

70/   Improving  Congressional  Control  Over  the  Federal  Budget,  op.  cit.:  11-12. 
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1 .   1:  is  learr.ir.g  zt.zz.   by  doing.  .  .  . 

1.   It  is  b-ilcitj:  at  interdisciplinary  staff.   Ftcti  a  ::::;;;::-=:  staff  larjelv 
c:  rersr-s  viz':.   bac'Kgrc urcs  it.  acocctcir.E  ace  :::'.;::::-  =  :  a.iitir.g.  :.-.:  is 
shifting  t:  a  staff  safe  uc  cf  nar.v  disciplines —  incl-ding  iz.zzz.eezs,    econo- 
mists, mathematicians,  statisticians,  actuaries,  systems  analysts,  persons 
vith  academic  or  experience  backgrounds  in  business  or  public  administration, 
and  a  sprinkling  of  exper--  in  other  fields.   Over  20  percent  of  Che  profes- 
sional staff  is  composed  of  persons  trained  in  disciplines  other  than  account- 
ing and  this  proportion  is  growing. 

3.   It  makes  extensive  use  of  consultants  in  various  fields  to  provide  expert 
advice  and  assistance. 

U.      It  contracts  vork  out  on  occasion. 

5.  It  conducts  extensive  training  rrc;r=-x=  and  holds  special  Srr-'-.ars  or:  orcgran 
evaluation  in  different  fields.   ...Seminars  have  been  held  on  health,  wel- 
fare, education,  at:  lav  en: croEnenc  assistance. 

6.  It  takes  advantage  of  on— going  or  completed  analytical  and  evaluation 
ycrk  cf  ether  "ederal  agencies  and  outside  ergsniz-atiens  ,  such  as  the  Verse 
Institute  and  the  Brookings  Ins  titration.   It  is  developing  a  guide  to  program 

evaluation  studies .... 

It    r.as    asssnrlsi    at    acTiscrv   canel   cf    a::_:    2'.    =~  =  zezLS    atd   tr:gra-    analysis 
specialists  vho   are  expert   in  numerous   fields  of  government  operations  and 

eval-atioc. ,  "- 

3.      Illustrations    ?rrvided   by    the    1-Al    cf    Its   ?rcgran   Evaluation   Activities. 
Assistant   Comptroller  General  Morse  has   identified  examples  of   different    —as    :f 
program  eval.acurs    :-e    age-.:  -■    has  t>erf  crmed.  "-/     One   cf    tcese    is    inserted  below  to 
illustrate   the   agency's   conception  of  its   activities   in   this   area: 

Improvement   of   definition  of  a  programs   goals    and    :'c;  actives 

Title  I  of   the  Elementary  and  Secondary  Education  Act  of  1965  authorized 
Federal  financial  assistance  to  local  educational  agencies  for  programs  to  help 
meet   special   educational  needs  of   deprived   children  in  areas  vith  high   concen- 
trations of  children  from  low- income  families.     The  Federal  Office  of  Education 
required  local  agencies  to  establish  measurable  objectives  and  related  pro- 
cedures to  evaluate  the  success  of  their  projects.     GAO's  audit  of  this  program 
in   Illinois   showed  weak  performance  on  this   score.      Objectives  stated  were 


71/     Horse,    Ellsworth  B.    Jr.    The  Auditor   Takes   a  Program  Evaluation.      The  Federal 
Accountant,   v.   XJJI,   no.    2,  June  1973:      b-~ .   Emphasis   in  original. 

22.1     Most  GAD  reports  are  public  information.      In  addition  short  abstracts   are 
published   in  annual  GAO  reports. 
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generally  vague  (e.g.,  build  a  varied  vocabulary)  and  were  not  stated  in  measur- 
able terms  by  types  and  degree  of  changes  sought  (e.g.  rate  of  increase  in  vocabu- 
lary). 73_/ 

4.  Other  GAP  Program  Evaluation  Activities.   GAO  has  also  published  two  reports 
which  deal  more  directly  with  improving  the  process  of  conducting  program  evaluations. 
These,  described  in  detail  in  other  sections  of  this  report,  consist  of  (1)  a  list 

of  legislative  requirements  for  program  evaluation,  including  steps  the  Congress 
needs  to  take  to  state  program  objectives  in  a  form  compatible  with  evaluation 
capabilities  and  (2)  an  inventory  of  agency  information  systems  which  include  program 
evaluation  reports  to  support  better  dissemination  of  program  evaluation  reports  to 
other  researchers  and  to  other  agency  administrators.   The  agency  is  now  preparing 
a  guidebook  of  recommended  standards  for  program  evaluation.   It  will  be  published  in 
1975. 

5.  Activities  and  Legislation  in  Support  of  Giving  GAO  More  Responsibility  for 
Program  Evaluation.   The  General  Accounting  Office's  program  evaluation  activities 
seem  to  have  Increased  considerably  since  passage  of  the  Legislative  Reorganization 
Act  and  the  Budget  and  Impoundment  Control  Act.   Several  factors  might  be  considered 
in  assessing  these  expanded  responsibilities  and  functions.   Probably  the  most 
important  are: 

-psychological,  organizational,  and  methodological  factors  which  surround  the 
evident  transition  in  the  agency  from  considering  evaluations  as  primarily  easy-to- 
measure  auditing  activities  to  conceiving  of  evaluations  as  harder-to-measure  achieve- 
ment, impact  and  performance  activities; 

-steps  the  agency  is  taking  to  enlarge  its  staff  to  perform  program  evaluations 
and  to  enlarge  its  staff  to  include  more  social  scientists  to  provide  the  inter- 
disciplinary mix  required  for  program  evaluation;  and 


73/  Morse,  op.  cit. 
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-the  time  lag  between  establishment  of  new  legislative  authority  and  the  estab- 
lishment of  effective  operational  procedures  to  meet  new  requirements. 

As  noted  above,  under  the  terms  of  the  Congressional  Budget  and  Impoundment  Act, 
passed  June  21,  1974,  GAO  was  gir'en  additional  authority  to  conduct  program  evaluation 
and  to  engage  in  other  activities  in  support  of  expanding  legislative  branch  program 
evaluation  capabilities.   The  agency  is  authorized  to  establish  an  Office  of  Program 
Review  and  Evaluation.   Among  its  functions  are  the  following: 

-to  review  and  evaluate  Government  programs  and  activities,  at  the  initiative  of 
the  Comptroller  General  or  at  the  request  of  either  House  or  Committees; 

-to  assist  committees  in  developing  statements  of  legislative  objectives  and 
goals  and  methods  for  assessing  and  reporting  actual  program  performance  in  relation 
to  such  legislative  objectives  and  goals;  and 

-to  recommend  methods  of  assessment,  information  which  should  be  reported,  respon- 
sibility for  reporting,  frequency  of  reports,  and  feasibility  of  pilot  testing;  and 

-to  review  the  program  evaluation  reports  and  activities  of  Federal  agencies. 74  / 


74/  Congressional  Budget  and  Impoundment  Control  Act,  Title  VII,  op.  cit, 
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PART  IV.   THE  PROCUREMENT  AND  CONDUCT  OF 
PROGRAM  EVALUATION  RESEARCH 

In  the  most  general  sense,  program  evaluation  is  a  form  of  applied  social 
science  research  which  uses  the  interdisciplinary  resources  and  techniques  of  economics, 
psychology,  sociology,  and  political  science.   Some  program  evaluation  research  is 
performed  in-house  by  social  scientists  on  an  agency's  staff.   These  evaluations 
generally  are  of  two  types:   (1)  auditing  activities  to  assess  the  allocation  and  use 
of  program  funds;  and  (2)  critical  syntheses  of  the  findings  of  the  evaluation  studies 
performed  for  the  agency  to  generate  recommendations  for  improving  the  administration 
and  operation  of  a  program  or  to  determine  whether  to  modify  or  terminate  the  programs. 

Evaluations  of  program  performance  or  of  the  effectiveness  and  impacts  of  a 
particular  program  intervention  strategy  are  performed  generally  on  an  extramural 
basis  by  social  scientists  or  interdisciplinary  program  evaluation  research  teams 
funded  by  the  agency.   The  Office  of  Management  and  Budget  has  defined  four  general 
types  of  evaluation  research,  delineated  on  the  basis  of  their  purpose  and  pre- 
dominant methodology.   These  generally  correspond  to  the  categories  used  by  Wholey, 
et.  al.  in  their  landmark  study  on  this  topic.75/  The  types  are  substantive  impact 
evaluations;  relative  effectiveness  evaluations;  process  or  management  evaluations; 
and  project  evaluations.   The  OMB  definitions  of  these  types  of  evaluation  are  given 
below: 

Substantive  impact  evaluations  attempt  to  measure  the  impact  which  federal 
programs  have  upon  their  stated  objectives.   This  type  of  evaluation  seeks  to 
determine  what  the  program,  accomplishes,  how  these  accomplishments  compare  to 
their  intended  purposes,  and  their  costs.   The  purpose  of  such  evaluations  is 
primarily  to  provide  information  for  use  in  major  policy  formulation. 

Relative  effectiveness  evaluations  seek  to  compare  the  effectiveness  of 
two  or  more  major  program  strategies  or  approaches  in  attaining  ultimate  objec- 
tives within  a  national  program.   These  studies  are  designed  to  help  policy 


75/  Wholey,  Joseph  S.,  et  al.  Federal  Evaluation  Policy:   Analyzing  the  Effects  of 
Public  Programs.   Washington,  D.C.,  The  Urban  Institute  [1970]  chap.  2. 
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officials  and  program  managers  select  the  most  effective  mix  of  services  to 
maximize  programs'  total  impact,  such  as  the  mix  of  skill  training,  remedial 
education,  and  job  search  assistance  in  a  manpower  program.   However,  these 
studies  do  not  necessarily  measure  the  impact  of  the  total  program  in  absolute 
terms  on  its  objectives. 

Process  or  management  evaluations  are  designed  to  measure  the  operating 
efficiency  of  national  program.   They  are  intended  principally  to  help  program 
managers  achieve  the  most  efficient  deployment  of  available  resources,  rather 
than  help  policy  officials  arrive  at  major  decisions  affecting  the  scope  and 
focus  of  the  national  programs. 

Project  evaluations  are  directed  to  individual,  locally  based  projects  which 
are  components  of  a  national  program,  regarding  the  impact  or  efficiency  of  the 
total  national  program.   Project  evaluations  may  entail  any  of  the  three  pre- 
ceding types  (substantive  impact,  relative  effectiveness,  or  process  evaluation) 
as  well  as  project  rating  comparing  the  effectiveness  of  one  or  more  individual 
projects  against  others.  j(J 

Four  types  of  research  performer  are  used  to  perform  extramural  evaluation  re- 
search activities.   The  first  two  are:   (1)  social  scientists  supported  by  grant  or 
contract  to  a  non-profit  making  institution;  and  (2)  social  scientists  at  federally 
funded  research  and  development  centers,  which  are  R  and  D  performing  organizations 
usually  created  and  exclusively  or  substantially  funded  by  one  or  more  Federal  agencies 
to  provide  the  agency  with  a  centralized  R  and  D  capacity  needed  to  develop  and 
evaluate  its  programs.   Typical  of  these  are  the  Urban  Institute  and  the  Institute 
for  Poverty  Research  at  the  University  of  Wisconsin,  whose  major  functions  were  to 
supply  expert  information  and  advice  on  urban  and  poverty  programs,  respectively.   A 
third  type  of  performer  consists  of  university-based  social  scientists.   The  fourth 
type  of  performer  is  social  scientists  or  interdisciplinary  teams  (including  operations 
researchers  and  systems  analysts,  etc.)  employed  by  profit-making  research  or  con- 
sulting firms  working  under  contracts  or  gr«nt6  awarded  on  a  competitive  or  non- 
competitive basis  by  an  agency. 

A.   Issues  Relating  to  the  Funding  of  Program  Evaluation  Research 

1.   The  Lack  of  Appropriate  Information.   The  executive  branch  does  not  maintain 


76/  Lewis  and  Zarb,  op.  cit.:  309. 
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an  inventory  describing  precisely  how  much  money  has  been  obligated  for  program 
evaluation  research.   Frequently  there  are  discrepancies  in  the  data  systems  used. 
For  instance,  Laurence  E.  Lynn,  Jr.,    former  Assistant  Secretary  for  Planning  and 
Evaluation,  Department  of  Health,  Education  and  Welfare,  noted  in  1972  that  "The 
Department  has  approximately  125  full-time  professional  evaluators  and  40  to  50 
million  dollars  available  for  evaluation  functions. "77/  According  to  a  report  cited 
by  Albert  D.  Biderman  and  Laure  M.  Sharp,  the  costs  for  total  HEW  evaluation  research 
amounted  to  $29.6  million  in  1971.28/ 

Annual  Budget  Appendix  volumes  sometimes  report  an  entry  for  program  evaluation. 
However,  this  item  is  not  reported  for  all  social  programs  listed  in  the  Catalog  of 
Federal  Domestic  Assistance.   For  instance,  the  Fiscal  Year  1974  budget  appendix  il- 
lustrates these  discrepancies  for  three  agencies  which  have  large  program  evaluation 
research  obligations:   (1)  Health  Services  and  Mental  Health  Administration,  lists  an 
item  for  research,  but  not  program  evaluation;  (p.  379);  (2)  Law  Enforcement  Assis- 
tance Administration  lists  have  no  entries  for  either  research  or  program  evaluation 
(p.  627);  (3)  Manpower  Administration,  Department  of  Labor,  lists  an  item  entitled 
Planning,  evaluation,  and  research,  (p.  631). 

Biderman  and  Sharp  attribute  the  dearth  of  information  on  the  funding  of  program 

evaluation  research  to  the  fact  that  existing  scientific  research  reporting  categories, 

including  those  used  by  NSF,  do  not  Include  the  classification  "program  evaluation 

research";  and  there  are  no  well-established  mechanisms  within  agencies  to  obtain  data 

on  the  funding  of  such  research: 

All  available  statistical  data  on  the  organization,  financing,  personnel 
and  functions  of  research  utilize  categories  that  have  no  close  correspondence 


77/   Lynn,  Laurence  E.  Notes  from  HEW.   Evaluation,  Fall,  1972:  26. 

78/  Citing:  James  G.  Abert.   Cost  and  Cost-Effectiveness.   Paper  delivered  at  Inter- 
national Federation  of  Operational  Research  Societies  International  Cost-Effec- 
tiveness Conference,  Washington,  D.C.,  April  12-15,  1971.  (Unpublished.)  From: 
Biderman,  Albert  D.  and  Laure  M.  Sharp.   The  Competitive  Evaluation  Research 
Industry.   Washington,  D.C.,  Bureau  of  Social  Science  Research,  Inc.  [1972]  p. 21. 
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with  the  world  of  SPER.*  The  primary  sources  of  data  on  American  research 
activities  are  oriented  to  the  academic  science  system  or  to  medical  and 
technological  research  and  development.   SPER  falls  neatly  into  none  of  these. 
That  most  used  source  for  charting  trends  in  American  research  activity,  the 
National  Science  Foundation's  series.  Federal  Funds  for  Science,  is  of  scant 
use  for  charting  trends  in  ^PER.   Even  if  cross-tabulations  of  data  of  this 
series  were  available  by  fiCJ.d  of  science,  funding  agency,  and  research  per- 
former, which  they  are  not,  we  would  have  no  way  of  knowing  which  table  cells 
would  include  SPER. 

While,  by  definition,  all  SPER  is  social  research,  not  all  of  it  is  reported 
as  social  science  research  and,  since  it  is  often  supported  from  operational 
program  funds,  an  unknown  proportion  of  the  total  is  not  reported  as  science 
or  research  at  all.   Further,  allocations  to  evaluation  work  are  frequently  made 
by  the  executors  of  federally-financed  action  programs  or  projects — state,  local 
or  private  agencies  which  may  be  layers  removed  from  the  point  at  which  book- 
keeping entries  are  made  that  are  reflected  in  NSF's  Federal  Funds  for  Science. 

Although  there  is  considerable  literature  arising  from  interests  in  govern- 
ment contracting,  including  studies  of  R  &  D  procurements  (Report  to  the 
President.  .  .  .,  1962),  studies  have  not  focused  on  what  until  recently  has 
been  that  relatively  tiny  part  of  all  procurement  which  involves  social  scientific 
or  related  work.   Other  than  those  dealing  with  contracting  for  developments  of 
defense  weapon  systems,  the  major  studies  have  dealt  primarily  with  the  National 
Aeronautics  and  Space  Administration  and  the  Atomic  Energy  Commission  (Senate 
Committee  on  Labor  and  Public  Welfare,  1965;  Orlans,  1967).  When  government 
financing  of  social  scientific  work  has  been  the  object  of  attention  of  studies 
(and  data  series),  the  preoccupations  have  been  almost  exclusively  with  the 
forms  of  financing  which  matter  most  for  institutions  of  higher  education  and, 
secondarily,  for  the  academically-oriented  institutes.   The  types  of  organi- 
zations which  perform  most  of  the  work  of  SPER  have  received  relatively  little 
consideration. 

In  the  recent  large  survey  to  examine  the  state  and  future  of  the  Social 
and  Behavioral  Sciences  (Behavioral  and  Social  Sciences  Study  Committee  of  the 
National  Academy  of  Sciences  and  the  Social  Science  Research  Council,  1969),  for 
example,  a  comprehensive  questionnaire  survey  was  carried  out  of  the  research 
finances  of  academic  departments,  professional  schools  and  university  institutes 
and  laboratories,  but  only  a  small  pilot  study  in  one  city — scarcely  analysed 
or  reported — was  performed  of  social  scientists  in  all  other  institutions.^ 

The  Survey  Committee  contracted  with  the  Bureau  of  Social  Science  Research  in 
Washington,  D.C.  for  a  questionnaire  survey  of  university  activities  in  the 
behavioral  and  social  sciences  which  were  reported  in  three  chapters  of  this 
report  (pp.  155-210).   A  small  pilot  survey  was  also  undertaken  by  the  committee 
in  the  Boston,  Massachusetts  area  to  study  the  employment  of  behavioral  and 
social  scientists  in  two  disciplines  (economics  and  psychology)  outside  the 
university  and  government;  this  was  reported  in  two  pages  of  the  report  (pp. 216- 
218).]   Such  studies  as  have  been  done  of  social  scientists  in  non-academic 
employment  (Radon,  1970;  Marcson,  1966)  have  involved  small  samples  including 
many  persons  working  in  non-research  roles,  so  that  it  would  be  impossible  to 
segregate  out  for  analysis  those  active  in  particular  research  pursuits,  such 
as  SPER._7_o7 
*  Social  Program  Evaluation  Research 


79/  The  Report  on  Survey  of  Federal  Program  Evaluation  Practices,  conducted  by  the 
staff  of  Senator  William  V.  Roth,  Jr.,  May  1972,  illustrates  the  difficulties  in 
trying  to  obtain  comparable  figures  from  Federal  agencies  even  when  a  well- 
formulated  questionnaire  is  used.   Quotation  from  Biderman  and  Sharp, op. cit.  p. 16-18. 
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B.  Difficulties  of  Obtaining,  Information  on  Types  of  Performers  of  Program  Evaluation 
Research 

One  of  the  most  important  implications  of  the  absence  of  information  is  that  it 
is  impossible  to  provide  precise  data  on  the  characteristics  of  performer,  that  is  in- 
house,  or  extramural  (either  profit-making  or  non-profit-making  research  organization). 
A  number  of  attempts  have  been  made;  they  have  involved  in-depth  surveys  or  collation 
of  scattered  in-house  report  forms. 

Most  of  these  studies  indicate  that  the  bulk  of  the  largest  awards  for  evaluation 
are  given  on  a  contract,  rather  than  a  grant  basis,  to  profit-making  or  non-profit- 
making  research  organizations  rather  than  to  university  based  social  scientists. 80/ 
This  finding  should  be  weighed  against  studies  which  indicate  that  evaluation  studies 
are  performed  best  by  social  scientists  at  universities  funded  on  grant,  rather  than 
a  contract  basis. 81/ 

For  example,  Abert  reports  that  about  45  percent  of  the  funds  HEW  awarded  in 
1971  for  program  evaluation  went  to  profit-making  research  organizations.   The 
recipients  of  other  funds  are  broken  down  as  follows:  not-for  profit  firms,  29  per- 
cent; universities,  21  percent,  state  and  Federal  government,  4  percent;  and  individual 
consultants,  1  percent. 82/ 

In  the  Spring  of  1974,  the  General  Accounting  Office  conducted  a  telephone 
survey  of  federally  funded  program  evaluation  on  behalf  of  the  Joint  Committee  on 
Congressional  Operations.   The  results  follow: 


80/  Biderman  and  Sharp,  op.  cit. 

81/  This  issue  was  discussed  for  example  at  the  "Dartmouth /OECD  Seminar  on  Social 
Research  and  Public  Policies,"  Dartmouth,  September  13-15,  1974,  especially  in 
a  presentation  by  Howard  Freeman,  based  upon  research  he,  Illene  Bernstein  and 
Patricia  Riecker  are  doing  for  the  Russell  Sage  Foundation.   Draft  report 
available  from:  Illene  Bernstein,  Department  of  Sociology,  University  of 
Indiana,  and  Dr.  Freeman,  director,  Institute  for  the  Social  Sciences,  UCLA. 
For  a  short  summary  of  the  report,  see:  Evaluation  Revisited,  Behavior  Today, 
May  27,  1974:  p. 148. 

8?/  Abert  in  Biderman  and  Sharp,  op. cit.,  p. 23. 
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The  response  to  our  survey,  which  are  summarized  in  the  following  table  were 
for  the  most  part  carefully  researched  and  prepared  by  the  agencies  listed.   A 
few  of  the  agencies  responded  in  writing-DOT,  Agriculture,  0E0,  VA,  and  the 
Special  Action  Office  for  Drug  Abuse  Prevention.   Problems  with  the  definitions 
of  program  evaluation  were  fewer  than  we  expected.   We  believe  the  total  estimate 
of  $130  million  budgeted  for  program  evaluations  in  Fiscal  Year  197A  is  fairly 
accurate  for  those  agencies  listed,  given  the  limitations  of  a  telephone  survey. 

PRELIMINARY  ESTIMATES  OF  FUNDS  BUDGETED  FOR  PROGRAM  EVALUATION,  RESPONDING  AGENCIES,  FISCAL 

YEAR  1974 

(In  thousands  of  dollars) 

Amounts  budgeted  for  evaluation  breakdown  of 
total  by- 


Agency 


Department  of  Health,  Education  and  Welfare  50r,  000  10,000  60,000 

Department  of  Transportation  7,095  18,875  25,970 

Environmental  Protection  Agency  5,756  6,667  12,423 

Department  of  the  Interior  3,800  7,584  11,384 

Department  of  Labor  8,900  567  9,467 

Department  of  Commerce  1,031  4,520  5,551 

Law  Enforcement  Assistance  Administration  4,337  165  4,502 

Veterans'  Administration  38  3,154  3,192 

Department  of  State  (except  AID)  13  3,091  3,104 

Department  of  Agriculture  2,000  1,075  3,075 

Office  of  Economic  Opportunity  1,550  1,023  2,573 

Department  of  Housing  and  Urban  Development  1,300  327  1,627 

Agency  for  International  Development  528  1,098  1,626 

Special  Action  Office  of  Drug  Abuse  Prevention  869  653  1,522 

Total  87,217  58,799  146,016 

Percent  of  total  59.7  40.3 


Agencies  unable  to  give  a  complete  response  in  the  time  allotted  for  this 
preliminary  telephone  survey:  Department  of  Defense,  Department  of  Justice  (Except 
LEAA),  Department  of  Treasury,  National  Aeronautics  and  Space  Administration. 83/ 

The  agency  did  not  give  a  breakdown  by  types  of  contract  performer  (that  is  profit 
or  non-profit  making).   It  seems  evident  that  a  continuing  data  system  be  established 
to  collect  information  on  federally-funded  program  evaluation  and  that  it  also  cate- 
gorize the  type  of  performer. 


8^   In  U.S.  Congress.  Joint  Committee  on  Congressional  Operations.   Congressional 

Research  Support  and  Information  Services:  A  Compendium  of  Materials.   Washington, 
U.S.  Govt.  Print.  Off.  [May  1974]  p. 290. 
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C.   Specific  Issues  Relating  to  Federally  Funded  Research  and  Development  Centers 
as  Performers  of  Program  Evaluation  Research 

The  Congress  has  not  given  in-depth  examination  to  the  conduct  of  social  pro- 
gram evaluation  research  by  federally  funded  research  and  development  centers  (FFRDC). 
It  is  difficult  to  generalize  about  these  activities,  except  to  say  that  some  agencies 
have  used  them  more  than  others;  and  that  the  use  of  such  centers  in  the  social  policy 
area  is  probably  declining. g^/     Some  information  on  the  pros  and  cons  of  the  use  of 
FFRDCs  was  provided  in  a  1971  report  by  the  National  Academy  of  Sciences.   A  special 
advisory  committee  reported  to  the  Academy  on  the  operations  and  implications  of  one 
such  center,  the  Institute  of  Poverty  Research,  University  of  Wisconsin,  sponsored  by 
the  Office  of  Economic  Opportunity.   The  findings  of  this  study  are  specific  to  this 
particular  center  but  could  characterize  the  activities  of  other  centers. 

According  to  the  Committee,  policymakers   should  consider  a  number  of  issues 
which  surround  the  conduct  of  program  evaluation  research  of  this  nature: 

-  the  "danger  of  overselling  the  fruits  of  even  well-conducted  evaluations 
and  and  field  experiments;" 

-  the  "strong  partisan  political  currents"  that  interact  with  the  conduct 
and  use  of  reports  prepared  by  such  centers; 

-  the  need  for  replication  of  experiments  initially  conducted  by  the  center; 

-  a  "division  of  responsibility  for  the  conduct  of  an  evaluation  or  ex- 
periment between  two  or  more  teams  so  that  the  knowledge  of  each  would  serve  as 
a  check  against  the  oversight  or  misjudgments  of  the  others"; 


The  Office  of  Education  made  extensive  use  of  such  centers;  they  were  phased 
out  at  the  end  of  the  fiscal  year  1972.  During  the  period  1971-1973  HEW, 
exclusive  of  the  Office  of  Education  utilized  17  such  centers;  during  the 
period  1972-1974,  the  agency  funded  only  13  centers.   (U.S.  National  Science 
Foundation,  Federal  Funds  for  Research,  Development,  and  Other  Scientific 
Activities,  Washington.  U.S.  Govt.  Print.  Off.,  1972,  p.  46.   (NSF  72-317); 
and  U.S.  National  Science  Foundation.  Federal  funds  for  Research,  Development, 
and  Other  Scientific  Activities.  Washington,  U.S.  Govt.  Print.  Off., 
1974.  pp.  51-52.   (NSF  74-300). 
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-  the  need  for  "open  access  to  the  data  for  subsequent  recalculation,  criticism 
and  evaluation  by  independent  groups  as  required;" 

-  the  need  to  time  appropriately  General  Accounting  Office  and  executive 
agency  investigations  of  a  center's  activities  or  research  progress  and  preferably 
delay  such  reviews  until  an  evaluation  is  complete,  since  "the  threat  of  [un- 
timely] investigations  of  this  sort... might  discourage  investigators  of  high 
caliber  from  involving  themselves  in  such  policy-related  work;" 

-  the  need  for  policy  makers  at  all  levels  of  government  to  lower  expec- 
tations about  the  utility  of  policy-oriented  program  evaluation  research  since 
"social  science  research  is  able  to  provide  only  very  limited  illumination; "85 / 
and 

-  the  need  to  consider  a  number  of  trade-offs  between  the  advantages  and 
disadvantages  of  research  conducted  by  policy  institutes  and  to  provide  the 
institutes  with  "long-term,  relatively  unconstrained  funding"  since  Such  insti- 
tutes cannot  do  "first-rate  independent  research  [with]  hand-to-mouth  financing. "86/ 
The  Commission  on  Government  Procurement,  created  pursuant  to  P.L.  91-129,  to 

study  and  recommend  to  the  Congress  methods  "to  promote  the  economy,  efficiency,  and 
effectiveness"  of  procurement  by  the  executive  branch,  looked  at  the  procurement  of 
applied  social  science  research  and  program  evaluation  services.   The  report,  pre- 
sented to  the  Congress  in  December  1972,  recommended  that  FFRDCs  be  organized  and  used 

to  satisfy  needs  that  cannot  be  satisfied  effectively  by  other  organizational  re- 


85^  "While  social  science  can  help  in  the  design  and  improvement  of  programs,  good 
policy-making  requires  art,  judgment,  and  weighing  of  conflicting  values.   It 
can  never  be  reduced  to  [social]  engineering  in  the  familiar  sense  of  the  term." 
(Policy  and  Program  Research  In  a  University  Setting:  A  Case  Study.  Report  of 
the  Advisory* Committee  for  Assessment  of  University  Based  Institutes  for  Research 
on  Poverty,  Division  of  Behavioral  Sciences,  National  Research  Council.   [Wash- 
ington, D.C.,  National  Academy  of  Sciences,  1971]  p.  54.) 

86/   Ibid.,  pp.  47-55. 
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sources.   Any  proposal  for  a  new  FFRDC  should  be  reviewed  and  approved  by  the  agency 
head  and  special  attention  should  be  given  to  the  method  of  termination,  including 
ownership  of  assets,  when  the  need  for  the  FFRDC  no  longer  exists.   Existing  FFRDCs 
should  be  evaluated  by  the  agency  head  periodically  (perhaps  every  three  years)  for 
continued  needs. 87/ 

D.   Issues  Relating  to  Funding  Mechanisms  (Grants  and  Contracts)  in  Program  Evaluation 
Research 

1.   Issues  Relating  to  Profit-  Versus  Non-Prof  it-Making  Performers.   Albert  D. 
Biderman  and  Laure  M.  Sharp,  Bureau  of  Social  Science  Research,  are  conducting  an  in- 
depth  study  on  the  performance  of  program  evaluation  research  by  both  profit-  and  not- 
for-profit  research  institutions.  Their  interim  report,  dealing  primarily  with  profit- 
making  research  organizations  was  printed  in  1972.   Preliminary  findings  raise  a  number 
of  issues  resulting  from  the  conduct  of  social  program  evaluation  research  performed 
by  profit-making  organizations.   Generally  they  observe  that  competitive  bidding  for 
program  evaluation  research  through  the  request  for  proposal  (RFP)  system  is  waste- 
ful of  resources  and  deleterious  to  sound  research  since  a  large  number  of  organi- 
zations go  through  the  expensive  process  of  writing  unsuccessful  proposals.   They 
gave  an  example,  for  instance,  of  a  $4  million  procurement  for  which  research 
organizations  spent  $1.3  million  in  writing  proposals. 88/ 

They  also  cited  the  need  to  assess  the  disadvantages  of  the  ways  in  which  the 
RFP  system  tends  to  overly  prespecify  tasks  or  research  methodology  that  a  contractor 
should  follow. 89/  On  this  same  point  the  National  Advisory  Council  on  Education 


87/  Summary  of  the  Report  of  the  Commission  on  Government  Procurement.  (Not  the 
Official  Report.)  Washington,  U.S.  Govt.  Print.  Off.,  December  1972.  p.  46. 
For  a  detailed  description  of  the  Commission's  findings  on  performers  of  re- 
search and  development,  see  Chapter  3,  pp.  13-23  in  Report  of  the  Commission 
on  Government  Procurement,  Vol.  2  (Washington,  U.S.  Govt.  Print.  Off.,  1973). 

88/   Biderman  and  Sharp,  op.  cit.,  pp.  38-39. 

89, 

/  Ibid.,  passim. 
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Professions  Development  reports: 

Within  limited  evaluation  funding  on  the  whole,  funds  for  contract  research 
are  so  overbudgeted  and  in-house  staff  positions  and  funds  are  so  underbudgeted, 
that  good,  practical  conceptalization  and  -monitoring  of  evaluation  research  is 
thwarted.   In  some  cases  even  requests-for-proposals  must  be  developed  by  out- 
side contractors  before  proposals  for  research  can  be  called  for.   ...Requests- 
for-proposals  have  often  attempted  to  specify  the  evaluation  research  design 
rather  than  the  questions  of  or  issues  to  be  addressed,  or  the  final  character 
of  the  research  evidence. 90/ 

Other  issues  raised  by  Biderman  and  Sharp  and  the  Advisory  Council  are:  the  lack. 

of  time  agencies  give  to  writing  RFP's  since  many  put  out  calls  for  contact  program 

evaluation  research  only  a  few  weeks  before  the  fiscal  year  ends; 91/  monopolization 

of  contracting  by  a  few  profit-making  organizations;  overutilization  of  systems 

engineers  and  systems  analysts  to  perform  program  evaluation,  often  using  imprecise 

social  science  tools  and  techniques;  administrative  deficiences  in  profit-making 

organizations;  the  economic  efficiency  and  quality  of  the  product  obtained  from  the 

use  of  sole  source  contracts  vs.  the  competitive  bid  proposal;  and  cost  overruns  in 

social  R  and  D  activities  performed  by  not-for-profit  organizations . 92 / 

E.   Legislative  Requirements  to  Mandate  a  Certain  Percentage  of  Funding  for  Program 
Evaluation 

Another  funding  issue,  which  seems  to  warrant  resolution,  is  that  of  legisla- 
tively mandating  how  much  an  agency  should  spend  on  program  evaluation.   There  are 
two  prevailing  points  of  view  on  this  issue*   One  is  to  continue  the  practice  of 
having  the  Congress  authorize  a  fixed  percentage  of  appropriations  of  a  program  for 
evaluation  purposes.   A  number  of  suggestions  have  been  made  about  what  percentage 
of  funds  should  be  allocated  for  this  purpose.   In  an  interview  given  shortly  before 


90/   National  Advisory  Council  on  Education  Professions  Development.   Search  for 
Success:   Toward  Policy  on  Educational  Evaluation.   A  report  of  the...,  June 
1974.   Washington,  D.C.,  National  Advisory  Council  on  Education  Professions 
Development,  op.  cit.  p.  20. 

91/   Ibid.,  p.  20. 

92/   Biderman  and  Sharp,  op.  cit.,  passim. 
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he  left  office  as  Secretary  of  Health,  Education  and  Welfare,  Elliot  Richardson 

suggested  that  an  allocation  of  .5  percent  of  appropriations  for  a  program  would  be 

adequate. 93/   In  their  study  of  Federal  Evaluation  Policy,   Wholey  et.  al.,  suggested 

that  the  fixed  percentage  should  vary  depending  on  the  substance  of  the  program: 

Funding  levels.   A  reasonable  evaluation  budget...,  this  study  suggests  is 
likely  to  range  from  0.5  percent  to  2  percent  of  the  total  program  budget. 
[For  programs  which  do  not  already  include  a  fixed  percentage  for  evaluation] 
it  would  appear  reasonable  ...  to  allocate  up  to  2  percent  of  manpower  program 
funds,  up  to  0.5  percent  of  vocational  educational  funds,  and  up  to  0.5  percent 
of  Title  I  elementary  and  secondary  education  funds  to  program  evaluation. 94/ 

In  his  interim  report  to  congressional  committee  chairman,  on  "Legislative 

References  to  Evaluation,  1967-1972,"  August  11,  1972,  the  Comptroller  General  expressed 

another  point  of  view  —  that  legislation  should  not  specify  what  percent  of  the 

appropriations  should  be  devoted  to  evaluation.   His  statement  follows: 

...[We]  question  whether  a  legislative  requirement  that  a  fixed  percentage  of 
program  funds  or  a  fixed  dollar  amount  to  be  spent  for  program  evaluation  is 
wise,  particularly  where  the  agency  may  not  be  in  a  position  to  spend  the  money 
effectively.   Rigid  personnel  regulation  imposed  by  the  executive  branch  can  be 
a  contributing  factor;  scarcity  of  trained  personnel  has  undoubtedly  been 
another  factor.   [In  our  studies  of  evaluation  for  the  Congress] .. .we  will  give 
special  attention  to  this  [issue]  in  those  cases  where  Congress  has  enacted 
explicit  requirements  for  program  evaluation. 95/ 

F.   Alternative  Institutions  for  the  Conduct  of  Program  Evaluation  Research  Under 
Federal  Agency  Auspices 

Although  there  are  no  precise  figures  on  the  composition  of  performers  of  social 
program  evaluation  research  for  the  Government,  it  seems  reasonable  to  estimate  that 
about  half  of  this  research  is  performed  by  profit-making  research  groups  or  con- 
sulting firms,  approximately  30  percent  by  non-profit  institutions  and  20  percent  by 


93/   Conversational  Contact .. .with  Elliot  L.  Richardson.   Evaluation,  v.  I,  no.  1 
1972:   15. 

_9V  Federal  Evaluation  Policy,  op.  cit.,  p.  82. 

95_/  Letter  prefacing  GAO  Report  B-161740,  August  11,  1972. 
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universities. 96/  As  noted  throughout  this  report,  many  criticisms  have  been  raised 
about  the  uneven  quality  of  evaluation  research.   These  deal  especially  with  method- 
ological weaknesses,  the  tenuous  conceptual  link  between  program  evaluation  research 
design  and  findings  on  the  one  hand,  and  understanding  of  the  basic  dimensions  of 
human  behavior  on  the  other,  and  insensitivity  to  the  policy  and  political  constraints 
which  impact  upon  the  initiation  and  implementation  of  program  evaluation  reports. 
These  criticisms  are  levied  at  both  university  and  non-university  performers  and 
also  at  in-house  and  extramural  performers. 97/  Program  evaluations  by  non-university 
researchers  are  generally  timely,  but  weak  in  theory,  conceptual  framework  and  method- 
ological underpinning.  Those  performed  by  university  researchers  are  criticized  for 
being  overdue  and  insensitive  to  policy  implications.   There  seems  to  be  a  consensus 
in  much  of  the  recent  literature  that  the  conduct  and  utility  of  program  evaluation 
research  would  be  improved  if  the  Government  adopted  a  formal  policy  to  institution- 
alize a  sustained  program  evaluation  capability  in  research  organizations  which  have 


96/   Extrapolated  from  a  classification  of  performers  of  HEW  evaluation  research,  1971 
(The  Competitive  Evaluation  Research  Industry,  op.  cit.:   23.)   For  further 
details  on  funding  and  performers  see  the  draft  reports  of  Freeman,  and  Freeman, 
Bernstein  and  Rieker,  op.  cit.:  passim. 

97/  Evaluation  magazine  contains  several  articles  on  this  topic: 

"On  the  question  of  'contract'  evaluations,  Garth  Buchanan,  Pamela 
Horst,  and  John  Scanlon  vote  no — that  the  costs  of  unsatisfactory  evaluations 
produced  by  contract  are  far  more,  really,  than  the  high  dollar  cost  of  in- 
house  evaluation  efforts.   In  Evaluation's  interview  with  him,  Bertram  Brown 
votes  yes,  because  the  use  of  outside  evaluators  has  modified  the  impact  of 
administrators  who  want  'positive'  funding  attached  to  evaluations.   Howard 
Davis,  Carol  Weiss,  and  Lawrence  Lynn,  Jr.  are  among  others  who  suggest  that 
administrators'  desire  for  positive  findings  has  been  known  to  lead  to 
positive  findings,  but  Weiss  is  also  quick  to  point  out  that,  whatever  the 
pluses  or  minuses,  the  instability  of  most  evaluation  efforts  and  the  turn- 
over of  evaluation  staffs  is  bound  to  lead  to  the  use  of  consultants  in  one. 
capacity  or  another." 
(See  Evaluation,  v.  no.  2,  1973,  passim.) 
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a  strong  social  science  base .98  / 

Recommendations  for  improving  the  institutionalization  of  program  evaluation 
research  take  two  forms.   One  is  that  the  Government  should  support  the  creation  of 
research  units  designed  explicitly  to  provide  short  and  long-range  program  evaluation 
research  for  a  particular  agency  or  class  of  problems.   The  second  is  a  call  for 
giving  more  sustained  funding  to  university  researchers  to  enable  them  to  develop  a 
better  capability  to  do  basic,  but  policy-oriented,  social  science  research. 

Proposals  for  creating  research  units  to  provide  evaluation  research  for  a 
particular  agency  or  class  of  problems  call  for  institutions  of  varying  configuration. 
However  they  could  generally  be  characterized  as  large-scale  non-profit  applied  social 
science  organizations,  either  independent  of  an  agency  or  attached  to  a  Federal  agency 
like  the  recently  created  National  Institute  of  Education,  or  funded  by  an  agency  to 
provide  it  with  information  and  analysis,  like  the  Urban  Institute,  created  first  to 
serve  the  Department  of  Housing  and  Urban  Development.   This  section  summarizes  several 
of  the  published  arguments  for  the  creation  of  such  units. 

Hendrik  D.  Gideonse,  director  of  program  planning  and  evaluation  of  the  National 
Center  for  Educational  Research  and  Development  of  the  U.S.  Office  of  Education, 
proposed,  in  1970,  that  programs  for  social  services  would  be  improved  by  the  devel- 
opment of  national  research  policies  and  national  research  organizations  to  provide 
information,  analysis,  and  program  evaluation  research  for  these  programs.  Many  of 
the  propositions  he  set  forth  for  a  national  research  strategy  in  education  contain 
elements  which  can  be  generalized  to  other  areas.   His  points  are  summarized  below: 
-The  funds  a  nation  spends  for  educational  research  and  development  should 

be  considered  a  part  of  the  nation's  general  expenditures  for  scientific  research 


98/  For  example,  see  the  report  of  the  National  Advisory  Council  on  Educational 

Professions,  Search  for  Success,  which  recommended  steps  to  improve  educational 
evaluation  research.   Also  described  in  Evaluation  Reforms  Urged.  Behavior  Today, 
v.I,  no. 2,  August  5,  1974:1. 
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and  development.  Thus,  educational  research  funds  should  he  subject  to  the  same 
review  procedures  used  for  other  types  of  scientific  R&D.  Funds  for  education 
should  compete  with  funds  allocated  for  other  areas  of  R  &  D;  and  priorities  for 
educational  R&D  should,  therefore,  be  established  on  a  nationwide  basis. 

-Educational  R&D  should  be  interdisciplinary  (incorporating  research  into 
educational  technology,  social  change,  quality  of  instruction,  goals  of  learning, 
etc.);  thus,  such  research  should  be  conducted  by  interdisciplinary  teams  in 
large-scale  research  organizations  .99/ 

-Two  factors  now  discriminate  against  the  conduct  of  effective  educational 
research  by  university  based  social  scientists:   the  reward  structure  of  academic 
departments  and  self-generated  professional  values. 

-Goals  and  priorities  for  education  should  incorporate  many  diverse  values 
and  points  of  view,  not  only  those  held  by  social  scientists  or  educators;  there- 
fore national  research  strategies  should  incorporate  consideration  of  these  other 
factors. 

Gideonse  stressed  that  development  of  the  then  proposed  National  Institute  of 
Education  would  meet  these  criteria  since  it  would  coordinate  the  educational  research 
activities  of  several  agencies.   However,  he  cautioned  that  the  agency  would  have  to 
take  steps  to  meet  another  criterion  of  effective  educational  research  units.   It 
would  have  to  "build  the  political  decision  structures  which. . .constitute  the  necessary 
enabling  condition  for  success. "100/ 


99/  For  background  information  on  the  complexities  of  conducting  interdisciplinary 
social  science  research,  see:   U.S.  Congress.  House.   Committee  oh  Science  and 
Astronautics.   Subcommittee  on  Science,  Research  and  Development.   Interdis- 
ciplinary Research — An  Exploration  of  Public  Policy  Issues.  A  study  prepared 
by  the  Science  Policy  Research  Division,  Congressional  Research  Service, 
Library  of  Congress,  October  30,  1970.  Washington,  D.C.,  U.S.  Govt.  Print.  Off. 
1970.   106  pp.   ICommittee  Print.] 

100/  Gideonse,  Hendrik  D.  Policy  Framework  for  Educational  Research.   Science. 
December  4,  1970:   1058. 
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Peter  H.  Rossi  outlined  a  variant  of  this  position  when  looking  at  the  need  to 
create  non-profit  interdisciplinary  research  units  at  universities  or  research  units 
under  a  university  consortia  mechanism.   He  noted  that  program  evaluation  research 
required  input  from  a  number  of  disciplines  and  used  a  number  of  techniques.   Such 
research  is  expensive  and  requires  economies  of  scrale.   Only  large  research  units, 
"with  a  director  of  research,  his  own  staff,... a  more  or  less  integrated  program  of 
research,"  and  drawing  upon  a  number  of  disciplines  would  be  able  to  provide  research 
needed  for  the  future.  101/  These  organizations  would  utilize  the  talents  of  academic 
researchers  and  thus  would  have  to  overcome  a  number  of  current  obstacles  to  large- 
scale  interdisciplinary  social  science  research;  and  social  scientists  would  have  to 
be  given  a  modicum  of  autonomy  similar  to  that  permitted  in  academic  departments. 
Teams  of  social  scientists,  with  appropriate  research  support  personnel  would  have  to 
be  created;  and  social  scientists  would  have  to  learn  to  work  within  the  political 
and  social  constraints  inherent  in  applied  social  research,  constraints  caused  by 
problem  definition,  the  conduct  of  research,  and  the  procedures  to  publish  and 
disseminate  research  findings. 102/ 

The  most  recent  proposal  along  these  lines  comes  from  the  Social  Science  Research 
Council's  Committee  on  Experimentation  as  a  Method  for  Planning  and  Evaluating  Social 
Intervention.   When  assessing  the  capability  of  existing  research  organizations  to 
do  policy  relevant  social  experiments,  the  committee  concluded:  "...these  existing 
organizations  possess  little  or  no  capability  to  design  experiments  and  manage  a 


101/  Rossi,  Peter  H.  Observations  on  the  Organization  of  Social  Research.   In 

Evaluating  Social  Programs:  Theory,  Practice,  and  Politics.   Edited  by  P.H. 
Rossi  and  Walter  Williams.  New\ork,  Seminar  Press,  1972.  pp.  267-87. 

102/  Ibid.,  p.  283. 
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flexible  field  organization. "103/  The  committee  recommended  an  organizational  locus 
for  such  studies:   the  provision  by  the  Federal  Government  of  relatively  small 
institutional  grants  ranging  in  size  from  $250,000  to  $500,000  per  year,  to  between 
six  and  ten  promising  research  institutions  or  a  consortium  of  institutions  to  enhance 
existing  capabilities. 

Walter  Williams,  a  social  scientist  with  long  experience  conducting  and  admin- 
istering program  evaluation  for  the  Government  described  the  rationale  for  this 
position  and  his  variation  of  the  recommendation.  Williams  states  that  the  Federal 
Government  should  allocate  sustained  institutional  support  for  this  purpose  to  social 
science  research  organizations  independent  of  universities.  104/  He  reasons  that 
university  based  social  science  research  organizations  are  incapable  of  meeting  the 
criteria  for 

...carrying  out... sound  large-scale  social  policy  research  and  policy  analysis 
[which]  seems  to  require  organizations  with  research  credibility,  staff  con- 
tinuity, a  reasonable  degree  of  control  over  the  allocation  of  staff  resources, 
a  strong  policy  commitment,  and  a  flexible  incentive  structure  that  rewards 
diverse  skills  in  terms  of  pay  and  status. 105/ 

University  based  social  scientists,  according  to  much  of  the  recent  literature, 


103/  Ouoted  in  Williams,  Walter,  The  Role  of  Social  Scientists  Outside  the  Government 
in  the  Federal  Social  Policy  Process,  a  background  paper  for  an  Address  to  be 
Delivered  on  August  25,  1974,  at  the  Canadian  Sociology  and  Anthropology  Association 
Meetings  in  Toronto,  Canada.   Draft:  p. 26.  The  committee's  full  report  is  available 
as:  Riecken,  Henry  W.  and  Robert  F.  Bourch,  eds.   Social  Experimentation:  A  Method 
for  Planning  and  Evaluating  Social  Intervention.   (Written  by  a  Committee  of  the 
Social  Science  Research  Council.)  New  York,  Academic  Press  [1974]  339  p. 

10V  Williams,  ibid.,  p.  36.   See  a  larger  exposition  of  this  position  in  his  "The 
Capability  of  Social  Science  Organizations  for  Performing  Large-Scale  Eval- 
uative Research,  in  Peter  H.  Rossi  and  Walter  Williams,  eds.   Evaluating  Social 
Programs,  Seminar  Press,  1972.  p. 306. 

105/  Williams,  The  Role  of  Social  Scientists  Outside  the  Government  in  the  Federal 
Social  Policy  Process,  op.  cit.,  pp.  22-23,  emphasis  in  the  original.  On 
this  point  see:   Coleman,  James.  Methodological  Principles  Governing  Policy 
Research  in-  the  Social  Sciences,  a  paper  presented  to  the  American  Association 
for  the  Advancement  of  Science,  December  29,  1972. 
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do  have  a  potentially  substantial  role  to  play  in  enhancing  the  capability  of  social 

program  evaluation  research  and  policy  analysis  for  the  Government.   According  to 

Williams,  "the  universities. . .probably  should  have  the  major  role  in  the  more 

fundamental  policy-oriented  research. . .[those] .. .more  basic  studies  {which]  offer 

the  traditional  incentives  of  the  past. .."106/  Carol  Weiss,  another  social  scientist 

with  extensive  program  evaluation  experience  has  elaborated  on  this  position: 

There  may  be  greater  potential  in  doing  research  on  the  processes  that  give 
rise  to  social  problems,  the  institutional  structures  that  contribute  to  their 
origin  and  persistence,  the  social  arrangements  that  overwhelm  efforts  to 
eradicate  them,  and  the  points  at  which  they  are  vulnerable  to  social  inter- 
vention.  Pivotal  contributions  are  needed  in  understanding  the  dynamics  of 
such  processes  and  in  applying  the  knowledge,  theory,  and  experience  that 
exist  to  the  formulation  of  policy. 107/ 


10fi/  Ibid.,  p.  37. 
102/  Weiss,  Where  Politics  and  Evaluation  Research  Meet,  op.  cit.,  pp.  44-45. 


560 


PART  V.   METHODOLOGICAL  ISSUES  IN 
THE  CONDUCT  OF  PROGRAM  EVALUATION 


A  number  of  issues  raised  in  previous  parts  relate  directly  to  the  general 
topic  of  this  section.   These  include,  for  example:  composition  of  the  inter- 
disciplinary team  conducting  the  program  evaluation;  types  of  performer; 
legislative  language  especially  in  relation  to  program  objectives,  requirements  for 
performing  evaluations,  legislative  statements  regarding  funding,  and  the 
compatibility  between  these  statements  and  program  evaluation  methodology;  proposals 
to  create  program  evaluation  information  dissemination  systems;  and  administrative 
obstacles  to  the  conduct  and  use  of  program  evaluation. 

Three  additional  issues  will  be  treated  here  since  they  are  receiving  a  good 
deal  of  attention  in  the  literature  on  evaluation.   They  are  the  appropriateness  of 
evaluation  techniques,  especially  social  experimentation,  improved  techniques  of 
data  collection  and  cross-fertilization  between  developments  in  social  indicators 
research  and  date  required  for  program  evaluation. 

Experimental  and  Non-experimental  Research  Design 

Three  basic  types  of  research  design  are  used  in  conducting  social  program 
evaluation:  non-experimental,  quasi-experimental,  and  experimental  designs.  Most 
federally-supported  program  evaluations  probably  use  non-experimental  research 
designs,  which  permit  two  types  of  evaluation  methodology:  before  versus  after 
program  comparison,  and  comparisons  of  planned  performance  versus  actual 
performance.   Non-experimental  evaluations  are  the  easiest  and  cheapest  to  conduct. 
However,  they  usually  are  not  planned  in  advance,  making  it  difficult  to  collect 
appropriate  data.   One  of  the  major  deficiencies  of  such  designs  is  that  they  fail 
to  control  for  many  of  the  observed  changes  which  occur  during  the  program's 
duration.   Thuss  it  is  difficult  to  explain  whether  the  program  intervention 
strategy  caused  the  changes,  if  any,  observed  in  the  participants,  or  whether  the 
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changes  were  caused  by  some  other  extraneous  factor  or  event  which  affected  the 
program  or  its  participants. 

Ouasi-experimental  designs  include  evaluations  which  project  pre-program  data 
versus  actual  post-program  data  over  a  period  of  time  or  compare  changes  in  program 
participants  with  jurisdictions  or  populations  not  served  by  the  program.   These 
designs  have  procedures  to  control  for  or  isolate  extraneous  factors  which  might 
influence  the  outcome  of  the  program  intervention  and  utilize  a  more  sophisticated 
data  base  and  logical  or  conceptual  underpinning  than  non-experimental  designs. 
However,  they  are  far  less  rigorous  and  explanatory  than  experimental  designs. 

Experimental  designs,  called  controlled  experimentation  or  social 
experimentation,  are  the  most  costly  and  difficult  types  of  research  used  in 
program  evaluation.   They  also  yield  the  most  reliable  and  valid  results.   However, 
they  require  mandatory  planning  before  the  program  is  implemented  in  order 
to  build  in  procedures  to  collect  appropriate  data  and  to  select  randomly  a  control 
group  and  test  population. 

See  the  attached  chart  for  a  short  description  of  evaluations  using  these  three 
types  of  research  designs. 

Scope  and  Limits  of  Social  Experimentation 

As  noted  throughout  this  report,  criticism  of  program  evaluation  and  its  use  in 
decisionmaking  is  widespread.   For  example,  Joseph  Wholey,  a  staff  member  of  the 
Urban  Institute,  and  widely  published  in  the  field  of  evaluation  research,  has 
summarized  some  of  these  criticisms  and  their  causes: 

Valid,  reliable  evaluation  is  very  hard  to  perform  and  can  cost  a  lot 
of  money.   Evaluators  have  real  problems  in  detecting  causal  connections  between 
inputs  and  outputs  and  in  doing  so  in  timely  enough  fashion  to  be  useful  to 
decisionmakers.   The  structure  of  a  program  can  have  an  important  influence  on 
the  technical  feasibility  of  separating  the  effects  of  the  program  from  the 
effects  of  other,  often  more  powerful,  forces  not  under  control  of  the  program. 

...Our  reviews  have  found  typical  Federal  program  evaluation  studies  marked 
by  certain  design  characteristics  which  severely  restrict  their  reliability 
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and  usefulness: 

(a)  They  have  been  one-shot,  one-time  efforts,  when  we  need  continuous 
evaluation  of  programs. 

(b)  They  have  been  carried  out  in  terms  of  national  programs  and  are 
very  weak  on  process  data. 

(c)  They  have  been  snail  simple  studies  working  with  gross  averages,  when 
we  need  studies  large  enough  to  allov  analysis  of  the  wide  variations 
we  know  exist  in   "*s  and  performance  among  projects  within  programs. 

These  studies  have  cften  beer  accompanied  by  conclusions  and  recommendations 
based  on  (insupportable  or  unmeasured  assumptions  and  weak,  and  often 
confusing  data.   In  these  cases,  the  evaluation  results  should  be  ignored  by 
policymakers.   Other  evaluation  studies,  while  competently  conceived,  are 
so  severely  constrained  by  tine,  money,  and  an  inadequate  data  base  that  the 
the  results  at  best  have  only  limited  significance  for  policy  changes  or 
program  improvement.  108/ 

".any  critics  of  current  progran  evaluation  practices  hold  that  this  situation 
would  be  improved  if  social  programs  requiring  evaluation  were  designed  and  implemented 
in  a  manner  which  permitted  evaluation  using  controlled  experiments  or  social  experi- 
ments.  For  instance,  the  Joint  Economic  Committee  conducted  a  series  of  investigations 
dealing  with  the  administration  and  evaluation  of  public  welfare  programs.   The 
difficulty  of  adhering  to  the  rigorous  criteria  required  in  social  experimentation 
-as  cited  as  a  principal  deficiency  of  these  programs: 

Improving  The  Evaluation  Process 

:ue  robust  expenditures  for  research  and  evaluation  of  training  programs 
(S179.4  million  from  fiscal  1962  through  1972)  are  a  disturbing  contrast  to 
the  anemic  set  of  conclusive  and  reliable  findings.  Although  some  of  the  data 
may  be  necessary  management  information,  much  of  what  Is  collected  as  a  matter 
of  course  by  prograr  administrators  cannot  be  used  to  estimate  the  impact  of 
training  and  determine  the  effectiveness  of  the  progTsm. 

Acccr.g  the  --  =  -.    glaring  deficiencies  are  ir.a"rcrriace  crr.trcl  etc.-s  ar.d 
short  observation  periods.   An  appropriate  control  group  is  essential  if  the 
impact  of  training  is  to  Ve  isolated  and  distinguished  from  the  influence  of 
other  factors.   Frequently,  studies  are  undertaken  without  any  control  group. 
When  a  control  group  is  Included  in  the  design,  it  is  almost  never  established 
at  the  time  that  enrollees  enter  the  training  program,   ^valuators  are  called 
in  after  training  has  occurred,  and  are  confronted  with  the  almost  impossible 
task  of  constructing  a  control  group  with  the  pretraining  characteristics 
and  experience  of  the  trainees. 

Observation  periods  rarely  last  more  than  a  year  and  usually  less.   This 
is  too  short  to  determine  how  long  benefits  last  or  whether  they  are  stable, 


10 8/  Wholey,  Joseph  S.   What  Can  We  Actually  Get  From  Program  Evaluation?  Policy 
Sciences,  v. 3,  September  1972:  36"-?. 
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increase,  or  decrease.   Evaluators  are  forced  to  base  their  estimates  of 
program  effectiveness  on  uneasy  assumptions  about  the  duration  of  earnings 
gains.   Tracking  trainees  and  a  control  group  over  an  extended  period  of 
time  is  an  expensive  proposition  but  it  is  probably  no  more  expensive  and 
certainly  more  useful  than  much  of  the  data  collection   and  evaluation  which 
has  been  conducted  in  the  past. 

Because  the  structure  of  programs-  and  the  characteristics  of  their 
clientele  change  over  time,  a  single  evaluation,  even  one  with  an  optimal 
design,  conducted  at  one  point  in  time  does  not  provide  reliable  information 
about  program  effectiveness.   A  standardized,  ongoing  evaluation  procedure 
should  be  estab  lishfd.  IK"/ 

Although  there  seems  to  be  growing  dissatisfaction  with  non-experimental 
program  evaluation  designs  and  considerable  enthusiasm  for  using  experimental 
designs  as  a  program  evaluation  strategy,  it  is  apparent  that  a  number  of 
expectations  and  concepts  of  evaluation  must  be  modified  before  this  approach  can 
be  used  successfully.   One  problem,  for  instance,  is  that  of  political  bureaucratic 
resistances  to  using  evaluation  of  any  sort  —  because  of  potential  negative 
consequences  for  an  agency's  or  a  committee's  pet  project.  111/ 

Researchers  have  given  considerable  attention  to  outlining  ways  to  overcome 
some  of  the  bottlenecks  which  hamper  the  utility  of  social  experiments  in  the 
policymaking  process.   Some  of  their  positions  are  summarized  below. 

Wholey,  for  instance,  notes  the  need  for  decisionmakers  to  be  aware  of  the 
requirements  of  experimentation:  continuous  evaluation,  better  data,  and  the  need 
for  larger  sample  populations;  and  the  need  for  evaluators  to  be  sensitive  to  the 
political  and  time  constraints  on  using  experimentation  as  an  evaluation  strategy. 
He  states  in  part: 

There  are  important  tensions  between  the  evaluator  and  the  program 


1  If/  U.S.  Congress.   Joint  Economic  Committee.   Subcommittee  on  Economy  in  Government. 
Economic  Analysis  and  the  Efficiency  of  Government.   Peport.   Washington, 
U.S.  Govt.  Print.  Off.   [February  9,  1970]  pp. 14-15. 

111/  See  for  instance,  Donald  T.  Campbell,  "Assessing  the  Impact  of  Planned  Social 
Change,"  Background  Paper  for  the  Dartmouth /OECD  Seminar  on  "Social  Research 
and  Public  Policies,"  September. 13-15,  J974.   passim. 
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official,  tensions  which  arise  out  of  the  very  notion  of  experimentation. 
The  criteria  for  selection  of  sites,  the  carefully  controlled  design  of  the 
experiments,  and  the  random  assignment  of  participants  (or  communities)  to 
treatments  are  basic  to  experimental  design.   The  program  administrator  may 
not  see  the  utility  of  such  ideas,  however.   What  is  so  wrong,  he  may  wonder, 
about  calling  an  existing  exemplary  program  an  "experiment"?  Or  why  not 
choose  the  people  most  in  need  of  housing  to  participate  in  a  housing 
allowance  experiment?  The  evaluator  must  woo  and  win  the  administrator  to 
the  need  for  preserving  the  experimental  character  of  the  experiment. 

Time  also  presents  an  enormous  problem  for  the  evaluator  of  experimental 
programs.   As  soon  as  there  is  sufficient  legislative  support  to  fund  a 
series  of  experiments,  there  may  be  enough  support  to  enact  such  a  program 
nationwide.   The  concern  that  legislation  will  be  enacted  before  the 
experiments  have  had  time  to  produce  reliable  results  nay  lead  to  pressures 
for  the  release  of  early,  less  reliable  findings.   The  New  Jersey  Income 
Maintenance  experiments  experienced  this  pressure.   Some  early  tentative 
results  from  the  study  were  released  with  reluctance  and  heavy  qualifications. 
If  experimentation  is  to  become  a  major  vehicle  in  policy  research,  then 
ways  must  be  found  to  anticipate  and  deal  with  these  types  of  pressures.  112/ 

Dr.  Alice  Rivlin,  senior  fellow,  Brookings  Institution  and  chairman  of  the 

National  Academy  of  Science's  Committee  on  Federal  Program  Evaluation,  is  one  of 

the  Nation's  foremost  authorities  on  using  social  experimentation  to  improve 

program  evaluation  and  ultimately  the  delivery  of  effective  social  services.   Her 

views  are  contained  in  a  recent  publication:  Systematic  Thinking  for  Social  Action, 

Washington,  D.C.,  The  Brookings  Institution,  Washington,  D.C. ,  1971,  150  pages. 

Rivlin  discusses  the  comparative  merits  of  several  programs  and  evaluation 

activities  of  them,  notably  the  New  Jersey  negative  income  tax  experiment  funded 

by  0E0,  the  follow-through  of  Project  Headstart,  and  performance  contract  teaching. 

Despite  her  advocacy  of  using  social  experimentation  as  a  tool  of  program 

evaluation ,  she  outlined  major  obstacles  to  its  use.   These  include:  changes  in 

objectives  of  the  program  as  it  is  administered;  the  need  for  objective  evaluation 

of  the  benefits  of  a  program;  problems  of  the  Hawthorne  effect  (that  is,  the  effect 

of  the  researcher  on  the  test  population  or  the  participants  of  the  program); 


Uj     Wholey.   What  Can  We  Actually  Get  From  Program  Evaluation?   Op.  Cit:  367-369. 
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generalizability  of  findings  from  a  selected  to  a  general  population;  political 
and  ethical  objections  to  experimentation;  and  problems  of  conducting  experiments 
and  evaluation  on  programs  whose  impact  might  not  be  immediately  noticeable.  113/ 

In  a  subsequent  analysis,  published  in  1973,  Rivlin  cautioned  that  social 
experiments  are  a  useful  tool  for  decisionmaking,  but  that  successful  social 
experiments  are  rare.   The  success  of  a  social  experimentation  strategy,  according 
to  Rivlin,  varies  with  the  type  of  program  intervention  strategy  employed.   Overly 
enthusiastic  advocates  of  experimentation  should  be  aware  that  this  is  not  an 
appropriate  evaluation  strategy  for  all  types  of  programs.   In  detail: 

It  seems  clear  that  experiments  are  feasible  when  the  treatment  is  a 
simple  one,  such  as  a  change  in  tax  or  a  payment  schedule  and  when  the 
outcome  of  interest  is  measurable  behavior  of  individuals,  such  as  hours  worked 
or  dollars  earned.   The  income  maintenance,  health  insurance,  and  housing 
demand  experiments  are  all  of  this  type.   They  pose  plenty  of  problems,  but 
they  certainly  can  be  carried  out. 

It  is  still  an  open  question  whether  or  not  more  complex  experiments 
are  feasible  —  those  in  which  the  treatment  is  a  multi-faceted  change,  such 
as  the  introduction  of  a  whole  new  education  curriculum,  or  where  the  outcome 
of  interest  is  a  change  in  a  whole  institution  such  as  a  school  system  or  a 
housing  market.   These  kinds  of  experiments  are  far  more  difficult  and  no  one 
is  sure  yet  that  they  can  yield  valid  results.  114/ 

"If  great  care  is  not  taken  to  make  current  experiments  as  useful  and  as 

sensible  as  possible,"  she  concluded,  "there  may  be  a  reaction  against  the  whole 

technique  and  a  potentially  useful  tool  may  be  taken  away."  115/ 


For  a  review  of  other  studies  treating  utilization  of  social  experimentation 
methodology  in  social  programs,  see:  Wegner,  Robert  E.  C.   Evaluating  the 
Evaluators.   Public  Administration  Review  [January/February  1973]   pp. 80-87. 
The  books  treated  in  addition  to  Rivlin's  are:  Kershaw,  J.  A.   Government 
Against  Poverty.   Chicago,  1970;  Levine,  R.  A.   The  Poor  Ye  Need  Not  Have  With 
You:  Lessons  from  the  War  on  Poverty.   Cambridge,  1970;  and  Williams,  Walter. 
Social  Policy  Research  and  Analysis.   New  York,  1971. 

Rivlin,  Alice  M.   Social  Experiments:  The  Promise  and  the  Problems.   Evaluation, 
v.I,  nc.?,    1973:  78.   In  this  same  respect  Walter  Williams  has  presented  a  strong 
case  for, using  a  policv  research  strategy,  rather  than  an  evaluation  strategy, 
to  serve  some  evaluation  requirements.   See  his:  The  Direct  Usefulness  of 
Evaluative  Research  and  Policy  Analysis  in  Social  Policy  Making,  an  address 
delivered  at  a  conference  on  "Evaluative  Concerns  in  the  Post-secondary 
Education  of  Students  from  Disadvantaged  Backgrounds,"  Mercy  College,  Detroit, 
June  22,  1974. 


115/  Idem. 
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Legislative  Interest  in  the  Ethics  of  Experimentation.   It  should  be  noted  that 
legislative  interest  in  the  ethical  issues  impinging  on  evaluation  and  experimentation 
is  indicated  by  the  passage  of  the  National  Research  Service  Award  Act,  P.L.  93-348, 
1974.   In  addition  to  other  functions,  the  Act  established  a  Commission  for  the 
Protection  of  Human  Subjects  of  Biomedical  and  Behavioral  Research  to  advise  the 
"  Department  of  Health,  Education,  and  Welfare,  by  identifying  ethical  principles  and  guide- 
lines which  should  underlie  the  conduct  of  biomedical  and  behavioral  research  involving 
human  subjects.   The  Commission  also  will  work  with  the  Department  to  develop 
administrative  procedures  to  carry  out  the  guidelines.   The  act  also  creates  a 
permanent  National  Advisory  Council  for  the  Protection  of  Subjects  of  Biomedical 
and  Behavioral  Research,  effective,  July  1,  1976;  and  requires  the  Commission  to 
undertake  a  special  study  of  the  ethical,  social  and  legal  implications  of  advances 
in  biomedical  and  behavioral  research  and  technology,  i  ifi/ 

Tbe  National  Institute  of  Mental  Health  awarded  $75,000  during  the  summer  of 
1974  to  the  Bureau  of  Social  Science  Research  to  develop  guidelines  for  proposed 
regulations  on  social  science  research  required  under  this  law.  jjj/ 

The  Need  for  Continuous,  Appropriate  Impact  Data 

One  of  the  most  frequently  voiced  complaints  made  by  program  evaluation 
researchers  is  the  absence  of  continuous,  appropriate  data  required  for  program 
evaluation.   As  noted  in  part  two  of  this  series,  much  of  the  reporting  information 
required  to  comply  with  administrative  requirements  for  Federal  grant  programs 


116/  Legislative  History:  House  Report  93-224,  May  23,  1973  to  accompany  H.R.  7724; 
Senate  Report  93-381,  August  3,  1973,  to  accompany  H.R.  7724;  Dates  of 
consideration  and  passage:  House:  May  31,  1<»73,  June  28,  1974;  Senate: 
September  11,  1973,  June  27,  1974. 

117/  Behavior  Today,  September  18,  1174:  236. 
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consists  of  program  output  information  stated  in  terms  of  dollars,  numbers  of 
people  affected,  and  the  like.  While  this  information  is  appropriate  to  admin- 
istrative categories  and  auditing,  it  is  not  useful  for  program  evaluation  since 
it  does  not  portray  harder-to-measure  performance  or  impact  characteristics.   For 
instance,  a  program  intervention  strategy  providing  education  to  train  unskilled 
workers  and  assist  them  to  find  employment  typically  requires  collection  of 
information  on  the  number  of  persons  who  obtained  jobs  after  the  program  or 
improved  scores  on  achievement  tests.   These  data,  which  may  be  collected  in  the 
course  of  normal  administration  of  the  program  are  not  necessarily  a  good 
indication  of  the  program's  success  and  failure  or  of  impacts  on  the  persons 
involved.  More  intangible  factors  and  information  may  have  to  be  considered,  such 
as  psychological  and  sociological  changes  in  program  participants  which  may 
change  outlooks  and  later  pay  off  in  job-hunting  and  employment  opportunities 
available  in  the  area. 

Although  generation  of  data  appropriate  to  measuring  a  program's  performance 
or  impacts  is  to  some  extent  the  responsibility  of  the  program  evaluator,  many 
administrators  and  evaluation  researchers  hold  that  procedures  should  be  built  into 
a  program  to  provide  for  continuous  collection  of  the  kind  of  data  required  for 
evaluation.   They  also  note  that  these  data  requirements  should  be  considered  by 
the  Congress  and  incorporated  into  authorizing  legislation  before  the  program  is 
implemented. 

Researchers  at  the  Urban  Institute  treated  this  issue  in  a  report  prepared 

for  the  Joint  Economic  Committee.   Excerpts  are  reproduced  below.  Manpower  training 

programs  were  the  immediate  issue: 

Manpower  Training  Information  Congress  Needs  and  Should  Require 

The , in format ion  about  the  cost-effectiveness  of  training  programs 
presently  available  has  been  found  wanting  in  consistency  and  usefulness. 
What  steps  can  Congress  take  to  assure  improvement  in  the  future?  To  answer 
this,  it  is  first  necessary  to  consider  the  several  facets  of  manpower 
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training  programs  over  which  Congress  may  exercise  its  review  powers.   These 
include: 

*  The  distribution  of  training  services  and  benefits  among  target  groups 
to  insure  that  national  distributional  goals  are  met. 

*  The  administration  of  the  national  programs  to  determine  whether  the 
the  programs  are  improving  over  time 

*  The  questions  of  costs  versus  benefits  to  determine  whether  the 
programs  are  worthwhile  investments  from  various  viewpoints. 

*  The  use  of  manpower  training  programs  as  economic  tools  in  dealing 
with  unemployment  and  inflation. 

The  most  valuable  information  for  the  purposes  of  the  Congress  may  well 
be  data  that  permit  comparisons  to  be  made  from  year  to  year  to  detect 
trends  in  identically  calculated  measures  of  the  programs.  To  improve  the 
accuracy  and  consistency  of  such  data,  the  Congress  should  require  that  infor- 
mation be  gathered  in  a  format  similar  to  that  shown  in  a  recent  Joint 
Economic  Committee  publication.  118/  The  format  for  distributional  data 
shown  there  should  be  adequate  and  could  be  provided  currently  by  the 
Department  of  Labor.   It  would  modify  the  cost  and  benefit  data  to  include 
for  each  program:  119/ 

*  Average  cost  per  enrollee. 

*  Number  of  enrol lees. 

*  Number  of  completers  (by  categories  in  those  programs  that  have 
multiple  categories  of  completers). 

*  Number  of  employed  completers. 

*  Average  workweek  of  an  employed  completer. 

*  Increase  in  income  of  employed  completer. 

All  benefit  data  should  be  accompanied  by  an  explanation  of  the  method 
of  determination  and  a  specification  of  the  groups  of  participants 
(completer,  employed  completer,  dropout,  etc.)  to  which  it  is  applicable. 

...We  recommend  that  unsupportable  assumptions  on  the  value  of  work 
performed,  social  costs,  private  costs,  or  projected  benefits  not  be  used  in 
the  benefit/cost  calculations.   However,  we  do  recommend  that  attempts  be  made 
to  measure  these  various  elements. 

Each  year,  the  Congress  should  also  request  the  Department  of  Labor  to 
examine  raw  cost  and  effectiveness  data  accumulated  at  the  appropriate 
administrative  level  for  all  projects  within  the  programs  and  use  these  for 
determining  the  causes  of  variations.  120/  These  data  could  then  serve  as  the 


118/  Economic  Analysis  and  the  Efficiency  of  Government.   Op.cit.,  appendix  3: 
Illustrative  Manpower  Overview  Data. 

119/  For  the  NYC  program,  measures  of  its  goals  of  keeping  participants  in  school 
would  also  be  needed. 

120/  For  programs  like  CEP  and  WIN,  the  appropriate  administrative  unit  would  be  the 
individual  CEP  or  WIN  project.   In  the  case  of  MDTA,  data  might  be  more  properly 
accumulated  at  the  State  level.   As  decentralization  proceeds,  the  relevant 
administrative  unit  might  become  the  local  prime  sponsor  of  each  comprehensive 
manpower  program.   This  should  improve  the  capability  for  review  both  in  DOL 
and  in  the  Congress. 
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basis  for  annual  discussions  between  Congress  and  the  executive  branch  on 
the  improvement  of  performance  within  each  program. 

If  the  information  provided  is  to  be  continuous  and  to  grow  in 
accuracy  and  consistency,  several  features  must  be  mandated  through  legis- 
lation. All  manpower  legislation  should  include  provision  for: 

*  One  standard,  comprehensive  system  of  required  reporting  from  the 
local  level. 

*  One-year  follow-up  c- .   the  wage  rate,  income,  and  job  stability  of 
participants. 

*  Systematic  processing  at  several  administrative  levels  to  provide 
plains  vs.  performance  information  and  relative  effectiveness 
information  on  local  projects. 

*  Reduction   of  this  information  to  the  formats  desired  by  the  Congress. 
...To  enable  more  knowledgeable  use  of  manpower  training  programs  as 

economic  tools,  there  is  great  need  for  validated  quantitative  models  that 
describe  the  impact  of  trainees  from  mjaipower  programs  on  local  labor  markets. 
It  is  time  to  stop  talking  about  what  effects  trainees  might  have  on  local 
markets  and  measure  what  these  effects  are  in  practice,  at  least  on  a  small 
scale.  Out  of  such  testing  could  come  more  definitive  advice  for  both  the 
Department  of  Labor  and  the  Congress  about  the  economic  effects  of  training 
programs.  We  recommend  that  Congress  direct  and  earmark  sources  for  data 
collection  projects  focusing  on  measurement  and  analysis  of  the  effects  of 
training  programs  on  local  labor  markets  under  different  conditions.  12J_/ 

Legislative  efforts  to  improve  the  collection  and  use  of  program  evaluation 

related  information  are  reflected  in  Title  VIII  of  P.L.  93-344,  the  Congressional 

Budget  and  Impoundment  Control  Act.  Under  this  Title,  the  Secretary  of  the  Treasury 

and  the  Director  of  the  Office  of  Management  and  Budget  are  to  cooperate  with  the 

Comptroller  General  and  the  Congressional  Budget  Office  in  developing  and  using 

budget  information  systems  which  include  "program-related  information."  Such 

budget  information  is  to  be  furnished  to  congressional  committees.   The  General 

Accounting  Office  is  also  authorized  to  develop,  establish,  and  maintain  an 

up-to-date  inventory  and  director  of  sources  and  information  systems  containing 

fiscal,  budgetary,  and  program-related  data  and  information  and  a  brief 


12V  Nay,  Joe  N. ,  Scanlon,  John  W.  and  Wholey,  Joseph  S.   Benefits  and  Costs  of 

Manpower  Training  Programs:  A  Synthesis  of  Previous  Studies  With  Reservations 
and  Recommendations.  In  U.S.  Congress.  Joint  Economic  Committee.  Benefit- 
Cost  Analyses  of  Federal  Programs.  A  Compendium  of  Papers.  January  2,  1973. 
Washington,  U.S.  Govt.  Print.  Off.,  1973:  pp. 259-61. 
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description  of  their  contents. 

Cross-fertilization  Between  Advances  in  Social  Indicators  Research  and 
Improvements  of  Data  Required  for  Program  Evaluation 

A  related  issue  is  that  of  the  relationship  between  social  indicators  and 

requirements  for  improved  data  for  program  evaluation.   Criteria  for  the  development 

of  refined  social  data  for  program  evaluation  purposes  closely  parallel  the 

objectives  and  expectations  of  social  indicators  research.  A  short  review  of  the 

relationship  is  given  below.   In  addition,  major  obstacles  to  the  development  of 

social  indicators  are  noted.   It  should  be  further  emphasized  that  many  of  the 

difficulties  of  developing  social  indicators  also  characterize  the  refinement  of 

program  evaluation  data. 

According  to  social  indicators  researchers,  social  indicators  would  overcome 

the  inadequacies  of  currently  collected  social  and  economic  data  and  would  provide 

for  the  measurement  or  quantification  of  qualitative  or  "soft"  social  factors.  122/ 

Current  social  indicator  research  activities  are  designed  to  develop  indicators 

which  would  reflect  the  following  properties: 

(1)   They  would  give  a  more  realistic  representation,  or  a  better  proxy  than 

do  current  data  series,  of  the  phenomena  or  causes  and  effects  which  they 

purport  to  represent.  For  instance  it  has  been  said  that  the  expansion 

of  food  programs  for  the  needy  does  not  adequately  indicate  whether 

hunger  and  malnutrition  are  being  eradicated.  Alternative  data  would 

describe  the  effects  of  such  programs  on  such  factors  as  improvements 

in  health  among  populations  served  by  food  distribution  and  funding 


122/  For  a  review  of  current  social  indicators  activities  and  an  enumeration  of 

policy  issues  relating  to  their  development,  see:  G.J.  Knezo.   Social  Indicators: 
A  Review  of  Research  and  Policy  Issues  (Including  Annotated  Bibliography). 
Science  Policy  Research  Division,  Congressional  Research  Service,  Library  of 
Congress,  Multilith  73-39  SP.,  February  9,  1973,  127  pp. 
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programs .  123/ 

(2)  They  would  differ  from  currently  collected  social  statistics  because  they 
would  measure  disaggregated  and  often  subjective  factors,  such  as  the 
amount  of  happiness  or  health  in  different  racial  or  ethnic  groupings 
scattered  throughout  the  country. 

(3)  They  would  be  normative  and  indicate  changes  in  the  quality  of  life.  The 
fact  that  attention  is  given  to  the  collection  of  data  about  the  purity 
or  impurity  of  the  air,  city  by  city,  or  region  by  region,  can  be  in- 
terpreted as  an  indication  that  the  society  collecting  this  information 
recognized  purity  of  air  as  a  social  goal. 

(4)  They  would  be  comparable,  longitudinal,  and  additive.  For  instance  the 
value  of  job  satisfaction  to  a  particular  individual  would  be  measured 
or  weighed  against  the  satisfaction  or  pleasure  he  derives  from  having 
ready  access  to  recreational  facilities.   Time  series  data,  or  data 
collected  over  a  number  of  years,  would  indicate  whether  one  or  another 
type  of  satisfaction  is  more  important  at  some  particular  time  such  as 
in  youth,  middle  age,  or  old  age. 

The  objective  of  developing  social  indicators,  with  these  properties,  is  to 


::r:_-i  ::  Dr.  '■ .    Mendelsohn,  M.rector  of  Ccnacunity  Pediatrics,  University  of 
Illinois  College  of  Medicine,  there  are  several  widely  used,  but  misleading, 
indicators  of  whether  hunger  and  malnutrition  are  being  conquered.  These  include: 
expansion  of  plans  for  food  stamp  distribution  programs;  increased  appropriations 
for  school  lunch  programs,  and  increased  funding  of  anti-poverty  programs.   As 
an  alternative  Dr.  Mendelsohn  contends  that  we  should  collect  and  measure  the 
possible  short  and  long-term  effects  of  such  programs.   These  new  data  series 
would  include  information  on:  decrease  in  infant  mortality  rates  between  families 
who  are  supported  by  these  programs  and  those  that  are  not;  differences  in  height 
butuuuB  piiwi If   7r:ETam  children  and  those  not  on  poverty  programs;  and  prevalence 
of  mental  retardation  between  those  on  and  off  poverty  programs.   (Testimony  of 
- ielsohn,  Select  Committee  on  Nutrition  and  Human  Needs.   In  Malnutrition 
and  its  Effects.   Statement  of  the  Honorable  Charles  Percy  on  the  Floor  of  the 
Senate.   Congressional  Record,   April  30,  1"70:  p. 6414.) 
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provide  program  administrators,  and  policymakers  with  more  accurate  and  valid  social 
data  to  draw  upon  in  evaluating  the  progress  of  social  programs  and  to  prepare 
.annual  balance  sheets  describing  the  social  state  of  the  Nation  and  alternative 
solutions  to  major  social  problems. 

An  indication  of  the  importance  of  this  issue  to  the  executive  branch  is 
indicated  by  publication  of  the  Office  of  Management  and  Budget's  first  social 
indicator  report,  Social  Indicators,  1973.  124/ 

It  should  be  pointed  out,  however,  that  there  is  a  continuing  debate  within 
the  social  science  communities  about  the  immediate  policy  utility  of  social 
indicators  research.   Some  social  indicators  researchers  hold  that,  imperfect  as 
they  are,  social  indicators  should  be  used  in  policy  formation.  125/  Others 
suggest  that  social  indicators  are  not  now  of  immediate  utility  to  policymaking, 
but  that  their  utility  would  be  enhanced  if  more  research  were  conducted  on 
subjective  dimensions  or  people's  perceptions  about  the  quality  of  life.  126 / 
Some  recommend  that  the  utility  of  social  indicators  in  policymaking  will  be 
hastened  if  greater  efforts  are  made  to  integrate  social  indicators  research 


124/  U.S.  Office  of  Management  and  Budget,  Statistical  Policy  Division  (with  the 
Social  and  Economic  Statistics  Administration,  U.S.  Department  of  Commerce). 
Social  Indicators,  1973:  Selected  Statistics  on  Social  Conditions  and  Trends 
in  the  United  States.  Washington,  U.S.  Govt.  Print.  Office,  1974.  258  pp. 

125/  For  example:  Taeuber,  Richard  C.   Social  Indicators  1973:  Some  Policy  Consider- 
ations.  Paper  presented  at  the  August  1974  meetings  of  the  American  Statistical 
Association,  12  pp.;  and  Mondale,  Walter  F.   Social  Accounting:  Evaluation  and 
the  Future  of  Human  Services.   Evaluation,  v.l,  1972:  30. 

126/  For  example,  see  the  reviews  in:  Parke,  Robert  and  Sheldon,  Eleanor  Bernert. 

Social  Indicators  One  Year  Later:  An  Overview.   A  paper  prepared  for  presentation 
at  the  meetings  of  the  Second  Annual  Social  Indicators  Conference,  sponsored  by 
the  Public  Policy  and  Issues  Division,  American  Marketing  Association, 
February  21,  22,  23,  1973.   Washington,  D.C.,  mimeo,  14  pp.;  and  Sheldon, 
Eleanor  Bernert  and  Parke,  Robert.   Social  Indicators:  Social  Science  Researchers 
Are  Developing  Concepts  and  Measures  of  Changes  in  Society.  Mimeo,  forthcoming 
in  Science,  34  pp. 
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arid  federally  sponsored  program  evaluation  activities.  127/  Some  researchers  have, 
in  fact,  faulted  the  0MB' s  first  effort  at  compiling  an  annual  social  report,  its 
study  Social  Indicators,  1973,  because  the  data  presented  were  not  interpretated  or 
or  evaluated  in  a  manner  which  gave  policymakers  any  indication  of  how  present 
social  conditions  or  changes  in  social  conditions  overtime  related  to  the  problems 
they  face.  128/ 

Some  recognition  of  the  need  to  develop  social  indicators  for  policy-making 
purposes  is  reflected  in  "The  Full  Opportunity  and  Social  Accounting  Act,"  which 
was  approved  in  the  Senate  in  both  the  91st  and  92nd  Congresses,  and  was  reported 
and  tabled  in  the  93rd  Congress.  129/  The  proposal  provided  for  creation  of  a 
Council  of  Social  Advisors  in  the  Executive  Office  of  the  President.   The  council 
"would  be  responsible,"  according  to  Senator  Walter  F.  Mondale,  the  bill's  prin- 
cipal sponsor,  "for  monitoring,  on  an  on-going  basis,  specific  actual  conditions 
in  the  country  which  affect  the  'social  opportunity'  of  our  people."  "Developing 
a  system  of  social  indicators,"  the  Senator  continues,  "would  be  a  principal  task 
of  the  council."  Utilizing  data  generated  by  the  Council  of  Social  Advisors,  the 
President  would  be  required  to  report  annually  "on  the  Nation's  social  status." 
The  following  areas  would  receive  priority:   "education,  health,  alienation, 
political  participation,  personal  security,  and  social  mobility."  130/ 


127/  Schneider,  Jerrold  E.  Making  Government  Work:  Political  Versus  Technical  Obsta- 
cles to  Social  Accounting.  American  Behavioral  Scientist,  v. 17,  n.4,  March/April 
1974:  585-607;  and  Mondale,  op.  cit. 

128/  Taueber,  op.  cit.:  6-7. 

129/  S.  5,  The  Full  Opportunity  and  National  Goals  and  Priorities  Act,  statement     . 
on  the  floor  of  the  Senate  by  Sen.  Walter  F.  Mondale.   Congressional  Record. 
January  4,  1973:  pp.  S52-56.   The  bill,  S.  5,  was  reported  by  the  Committee 
on  Labor  and  Public  Welfare  in  July  1973  (Sen.  Report  No.  93-324),  and  saw  floor- 
action  which  deleted  original  Title  II  of  the  proposal  which  proposes  to  create 
a  Congressional  Office  of  Goals  and  Priorities  Analysis. 

130/  Mondale,  Walter  F.  Social  Advisors,  Social  Accounting  and  the  Presidency.   Law 
and  Contemporary  Problems  [Duke  University  School  of  Law,  Summer  1970]:  497. 
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Such  a  system  is  necessary,  Senator  Mondale  says,  to  give  "societal  knowledge 
visibility  of  the  sort  that  only  presidential  involvement  can  generate,"  and  to  pro- 
vide Congress  and  the  President  with  better  data  on  which  to  make  more  informed 
decisions. 

Recent  literature  and  research  reports  on  social  indicators  have  identified 
a  number  of  other  obstacles  to  the  development  of  social  indicators  and  improved 
program  evaluation  data.   These  are  described  next. 

Understanding  of  cause  and  effect  relationships  underpinning  changes  in 
society  is  another  criterion  which  must  be  met  before  social  indicators  can  become 
useful  for  policymaking.   However,  according  to  Sheldon  and  Land,  "...the  state  of 
the  art  with  respect  to  the  development  of  explanatory  relationships  [in  the  social 
sciences]  is  especially  deficient. "131/ 

Another  of  these  problems  is  the  issue  of  invasion  of  privacy.   Longitudinal 
or  time-series  data,  in  some  cases,  requires  annual  interviews  of  the  same  persons 
on  particularly  sensitive  questions.   It  is  entirely  conceivable  that  individuals 
would  resist  questions  by  Government  officials  regarding  happiness,  marital 
problems,  and  the  like.   Furthermore,  objections  have  been  raised  in  many  quarters 
to  collection  by  Government  agencies  of  disaggregated  data  which  violate  protection 
of  fundamental  civil  rights.   A  case  in  point  is  the  disagreement  with  some  current 
Federal  agency  objectives  of  equalizing  employment  by  race,  religion  and  sex.   Some 
persons  object  to  the  establishment  of  goals  or  quotas  of  this  nature,  and  to  the 
collection  and  possible  evaluation  of  these  data. 

Hard-to-measure,  subjective  data  are  frequently  collected  by  means  of  survey 
research.   This  technique,  according  to  Albert  D.  Biderman,  has  serious  limitations 
which  should  be  remedied  before  it  can  be  employed  on  a  large  scale  in  social 
indicators  research: 


131/  Sheldon,  Eleanor  Bernert  and  K.C.  Land.   Social  Reporting  for  the  1970's: 
A  Review  and  Programmatic  Statement.   Policy  Sciences,  v. 3,  1972:  141. 
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...The  only  currently  available  means  of  securing  data  for  indicator 
systems  is  through  interview  and  questionnaire  surveys.  Although  the  Nation 
has  vast  resources  for  carrying  out  such  surveys,  only  a  small  proportion  of 
these  conform  to  the  exacting  standards  of,  for  example,  the  Census  Bureau. 
...Prevailing  standards  of  survey  practice  are  incompatible  with  meeting  even 
loose  criteria  of  validity...  I  cannot  see  the  goals  of  current  grant  proposals 
for  social  indicator  systems  being  met  without  considerable  qualitative  and 
quantitative  upgrading  of  the  survey  industry  ...132/ 

Notable  objections  have  also  been  raised  to  the  implementation  or  policy 

invocation  phase  of  social  indicators  activity.  Peter  Henroit,  a  widely  published 

student  of  social  indicators,  notes  that  more  and  better  social  information  does  not 

necessarily  lead  to  better  decision-making.   Political  decision-making,  based  upon 

bargaining  and  compromise  of  competing  interests,  occurs,  he  says,  regardless  of 

the  level  of  sophistication  of  data  generated  by  different  decision-making  bodies. 

As  a  case  in  point,  he  reports  that: 

Public  housing  was  placed  where  it  was  in  Chicago  primarily  because 
of  political  pressures  against  dispersion  and  no  amount  of  information  about 
social  costs  was  likely  to  have  made  any  difference.  133/ 

As  a  variant  of  this  caveat,  Frank  J.  Popper  has  warned  that  social  indicators, 

especially  the  choice  of  indicators  and  values  upon  which  they  were  based,  would 

lead  to  elitist  decision-making  far  different  in  tenor  from  the  democratic 

pluralism  characteristic  of  the  United  States.  As  an  illustration  of  this  notion, 

Popper  says: 

Toward  Balanced  Growth,  the  first  social  report  of  the  Nixon  Administration, 
avoided  discussing  such  issues  as  national  priorities,  race  relations,  urban 
decay  and  minority  dissent,  and  its  [discussion  on]  the  consumer  movement  did 
not  mention  the  activities  of  Ralph  Nader.  3-3V 


132/  Biderman,  Albert  D.   Social  Indicators  —  whence  and  whither.  A  paper 

prepared  for  presentation  to  the  First  Annual  Social  Indicators  Conference. 
American  Marketing  Association,  February  17-18,  1972.   [Washington,  D.C., 
unpublished]:   p. 17. 

133/  Henroit,. Peter  J.   Political  Questions  About  Social  Indicators.   The  Western 
Political  Quarterly,  v.  xxiii,  no. 2,  June  1970:  244. 

13V  Popper,  Frank  J.   The  Social  Meaning  of  Social  Accounting.   Polity,  v.4, 
.  Autumn. 1971:  86. 
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PART  VI.   THE  POSSIBLE  ISSUES  OF  LEGISLATIVE  CONCERN 
RAISED  BY  INFORMATION  PRESENTED  IN  THE  REPORT 

This  report  was  designed  to  provide  legislative  staff  with  background 
information  on  major  issues  of  current  concern  in  program  evaluation.   Summa- 
rized below  are  the  major  topics  addressed  and  the  possible  issues  of  legis- 
lative concern  generated  by  the  information  presented  in  the  report. 

I.   The  Initiation  and  Use  of  Program  Evaluation 

A.  The  Need  to  Clarify  Legislative  Statements  of  Program  Coals  and  the 
Purposes  of  Evaluation. 

1.  Consideration  of  the  need  for  the  Congress  and  for  Federal  agencies 
to  examine  the  merits  of  specifying  program  goals  and  objectives  more  pre- 
cisely to  promote  the  development  of  appropriate  yardsticks  or  goals  of 
programs  to  facilitate  the  conduct  of  program  evaluation. 

2.  Consideration  by  the  Congress  and  especially  by  authorizing  and 
appropriations  committees  of  the  need  to  clarify  and  further  detail  statu- 
tory requirements  for  program  evaluation. 

3.  Congressional  consideration  of  developing  appropriate  criteria 
and  expectations  for  the  performance  of  program  evaluation,  to  provide 
for  better  correspondence  between  existing  program  evaluation  techniques 
and  the  vague  aims  of  broad-scale  social  programs,  which  frequently  pre- 
clude adequate  evaluation. 

B.  Better  Communication  for  Effective  Evaluation. 

1.  Possible  inquiry  by  the  Congress  into  Federal  agency  and  social 
scientists'  techniques  and  obligations  to  provide  for  improved  communications 
between  evaluators  and  federal  administrators  to  eliminate  problems  encountered 
in  developing  and  conducting  program  evaluations. 

2.  Possible  congressional  inquiry  into  the  incentive  and  reward  system  of 
universities  which  tends  to  discourage  the  conduct  of  program  evaluations  use- 
ful to  policy-makers. 

C.  The  Integration  of  Program  Evaluation  Into  Program  Management. 

1.  Possible  congressional  inquiry  into  the  extent  to  which  agencies  use 
program  evaluation  in  planning  and  budgeting  and  of  ways  to  further  enhance 
the  utility  of  evaluation  reports  in  these  activities. 

2.  Consideration  of  the  need  for  agencies  to  develop  comprehensive 
evaluation  work  plans,  setting  priorities  for  program  evaluation  and 
recommending  program  evaluation  methodologies  most  appropriate  to  the  agency's 
objectives;  consideration  of  a  "preassessment  of  the  evaluability  of  programs." 

3.  Possible  congressional  encouragement  of  the  development  by  agency,  of 
program  evaluation  handbooks  for  use  within  the  agency  and  by  State  and  local 
administrators  who  are  given  greater  evaluation  and  accountability  respon- 
sibilities with  continuing  decentralization  of  some  grant  programs  and  revenue 
sharing. 
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4.  Possible  Congressional  inquiry  into  the  need  for  agencies  to 
establish  inventories  of  program  evaluation  research  in  process  and  finished, 
to  assist  agencies  in  eliminating  duplicate  program  evaluation  researches, 

to  determine  priorities  for  research,  and  to  assist  in  promoting  develop- 
ment of  an  overall  agency  evaluation  work  plan. 

5.  Consideration  of  requiring  Government-wide  inventories  of  completed 
program  evaluations  to  permit  sharing  of  program  evaluation  information 
gathered  with  Federal  funds,  to  enhance  the  evaluation  of  similar  programs 
in  other  agencies  and  geographic  locales,  and  to  stimulate  refinement  of  the 
methodology  of  program  evaluation. 

6.  Consideration  of  the  need  for  social  scientists  to  be  encouraged 
to  prepare  program  evaluation  reports  in  forms  most  compatible  for  policy- 
making purposes. 

D.   The  Need  for  Better  Dissemination  of  Program  Evaluation  Information 
Supported  Under  Federal  Agency  Auspices. 

1.  Consideration  of  inquiring  into  the  need  for  Federal  agencies 
to  encourage  program  evaluation  information  dissemination  systems  within 
agencies  to  assist  researchers  in  sharing  information  on  already  completed 
evaluations. 

2.  Consideration  of  Congressional  inquiry  into  the  need  to  establish  a 
Government-wide  inventory  of  abstracted  program  evaluation  reports  under  auspices 
of  an  executive  or  legislative  branch  agency  or  incorporated  into  existing  biblio- 
graphic devices  to  assist  researchers  in  sharing  information  and  to  expedite  the 
work  of  administrators  who  rely  upon  evaluations  in  assessing  the  merits  of  a 
Federal  program. 

Hi   Improved  Coordination  of  Program  Evaluation  in  the  Executive  and  Legislative  Branches 

A.  The  Need  to  Incorporate  Program  Evaluation  in  Executive  Branch  Budget  Processes. 

1.   Possible  Congressional  inquiry  into  steps  the  Office  of  Management  and 
Budget  should  take  to  further  encourage  the  use  of  program  evaluation  information 
in  preparing  budget  and  program  statements,  and  particularly  to  encourage  Federal 
agencies  to  collect  hard-to-measure,  but  relevant,  social  impact  data  in  the 
normal  course  of  administering  a  program  to  facilitate  evaluation  of  the  program. 

2.  Review,  as  needed,  by  the  Congress,  of  steps  the  OMB  is  taking 

to  utilize  program  evaluation  information  in  providing  for  a  better  evalu- 
ation of  how  alternative  programs  help  meet  national  needs,  and  related 
to  this,  the  evaluation  of  the  impacts  on  these  issues  of  the  OMB  re- 
organization and  "management  by  objectives"  effort. 

3.  Congressional  assessment,  as  needed,  of  program  evaluation  under 
the  revenue  sharing  program  especially  with  respect  to  the  effectiveness 
of  decentralized  accountability,  and  the  quality  of  auditing  and  evalu- 
ation standards  and  reporting  requirements  promulgated  by  the  General 
Accounting  Office  and  the  Offdee  of  Reven«e  Sharing  to  generate  appropriate 
evaluation  information. 
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A.   Consideration  by  the  Congress  of  proposals  to  give  OMB  in- 
creased responsibility  for  centralizing  and  overseeing  executive  branch 
program  evaluation  activities. 

5.  Possible  Congressional  examination  of  the  merits  of  the  recently 
proposed  Federal  Program  Evaluation  Office  in  the  Executive  Office  of  the 
President  or  in  the  Office  of  Management  and  Budget. 

B.  Issues  Relating  to  the  Improvement  of  Legislative  Branch  Program 
Evaluation  Capabilities. 

1.   Continued  congressional  consideration  and  oversight  of  ways  to 
fashion  a  better  link  between  the  process  of  congressional  oversight 
and  appropriations  to  mesh  legislative  review  and  evaluation  of  programs 
with  determination  of  appropriate  program  priorities  and  funding  levels. 

2.   Congressional  consideration  of  establishing  a  catalogue  which 
would  abstract  on-going  and  completed  program  evaluation  studies  by  agency 
and  social  program  area,  with  information  in  a  form  compatible  with  legis- 
lative information  requirements,  either  by  the  General  Accounting  Office, 
by  the  General  Accounting  Office  in  cooperation  with  the  Office  of 
Management  and  Budget,  or  by  a  separate  unit. 

C.  Issues  Surrounding  the  Role  of  the  General  Accounting  Office  in 
Providing  the  Congress  with  Improved  Program  Evaluation  Capabilities. 

1.  Continued  liaison  between  the  Office  of  Program  Review  and 
Evaluation  in  the  General  Accounting  Office  and  legislative  committees 
to  develop  statements  of  program  objectives  and  legislative  requirements 
for  evaluation  which  are  compatible  with  prevailing  techniques  for  the 
conduct  of  program  evaluation  and  the  collection  of  information  required 
for  evaluations. 

III.   The  Procurement  and  Conduct  of  Program  Evaluation 

1.   Possible  development  of  better  Government-wide  reporting  systems, 
by  the  National  Science  Foundation,  GAO  or  OMB,  on  the  conduct  and  utility 
of  program  evaluation,  requiring  submission  of  material  according  to 
uniform  categories  treating  funding,  performer,  type  of  evaluation,  purpose, 
and  methodology. 

2.  Congressional  inquiry  into  the  performance  of  program  evaluation 
research,  assessing  the  relative  merits  for  different  types  of  evaluation 
of  the  following  institutional  performers: 

-federally  funded   research  and  development  centers, 

-universities 

-in-house,  and 

-profit  vs.  non-profit  making  contractors. 

3.  Congressional  inquiry  into  problems  dealing  with  funding  mechanisms 
for  program  evaluation  research;  and  possible  revision  of  statutes,  as  suggested 
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by  the  Commission  on  Government  Procurement,  with  respect  to: 

-cost  overruns, 

-independent  research  and  development, 

-sole  source  contract  vs.  competitive  bidding, 

-monopoly  of  program  evaluation  contracting  by  a  few  industrial  firms, 

-the  merits  of  using  profit-making  vs.  non-profit-making  social  science 
firms  or  institutions  in  performing  program  evalution  research, 

-content  and  impact  of  poorly  stated  RFPs  for  program  evaluation  research,  and 

-proper  allocation  of  evaluation  functions  between  in-house  and  extra- 
mural performers. 

4.   Congressional  inquiry  into  the  efficacy  of  designating  appropriate 
levels  of  funding  for  program  evaluation  in  authorizing  legislation. 

A.   Alternative  Institutions  for  the  Conduct  of  Program  Evaluation  Research. 

1.   Possible  inquiry  into  the  merits  of  establishing  program  evaluation 
research  units  or  institutes  either  for  separate  agencies,  or  for  a  class 
of  social  programs  to  support  the  conduct  and  utilization  of  better  social 
program  evaluation  research. 

2.  Specifically,  congressional  and  Federal  agency  inquiry  into  the 
development  of  alternative  program  evaluation  institutions  which  would 
meet  the  following  concerns: 

-independence  of  administration, 

-bringing  a  national  perspective  rp  bear  in  development  of  goals  and 
research  strategies, 

-integration  of  policies  for  program  evaluation  research  into  other 
programs  and  policies  for  scientific  research  and  development, 

-linking  of  research  institutions  to  client  sponsors,  distributed  through- 
out the  country,  and 

-provision  of  adequate  training  and  facilities  for  interdisciplinary, 
policy-oriented  research. 

3.  Consideration  by  the  Congress  and  funding  agencies  of  developing 
policies  and  programs  to  promote  the  training  and  utilization  of  policy- 
oriented  social  science  researchers,  in  addition  to  discipline  oriented 
basic  and  applied  social  science  researchers. 

IV.   Methodological  Issues  in  the  Conduct  of  Program  Evaluation 

1.   Oversight,  by  appropriate  committees,  of  Federal  priorities  to  support 
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the  refinement  of  program  evaluation  methodology  for  use  by  Federal  agencies 
especially  programs  supported  by  the  Research  Applied  to  National  Science 
Foundation. 

2.    Consideration,  by  appropriate  legislative  committees  and  Federal 
agencies  of  the  pros  and  cons  of  social  experimentation  and  of  requirements 
for  program  evaluations  which  use  a  social  experimentation  methodology. 
Especially  relevant  here  are  the  issues  of  invasion  of  privacy,  ethical  and 
moral  questions  related  to  social  experimentation,  the  need  to  fashion  legislative 
requirements  compatible  with  the  conduct  of  social  experiments  (including  simulation 
of  "real  world"  conditions,  and  collection  of  appropriate  continuous  data). 

3.  Consideration  of  Congressional  attention  to  building  into  require- 
ments for  evaluation  of  data  collection  procedures  appropriate  for  program 
evaluation  techniques. 

U.      Consideration  of  Congressional  and  executive  branch  activities  to 
cross-fertilize  program  evaluation  and  social  indicators  research  to 
strengthen  the  development  of  both  areas  and  to  assist  in  providing  program 
evaluators  with  appropriate  and  continuous  valid  social  data. 
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Evaluations  of  Manpower  Training  Programs j 
Selected  References,  1970-1974.. 


A  well  developed  manpower  policy  is  concerned  with  facilitating  the 
arrangements  and  adjustments  of  the  labor  supply  to  the  labor  demand.  Employ- 
ment and  unemployment  statistics  provide  a  measure  of  the  success  of  the  overall 
policy. 

This  bibliography  focuses  on  one  facet  of  manpower  policy:  manpower  training 
programs  and  their  effectiveness.  For  the  purpose  of  the  bibliography,  manpower 
training  programs  are  defined  to  include  occupational  retraining,  public  employ- 
ment training,  vocational  rehabilitation  as  well  as  private  and  government 
sponsored  manpower  training  programs. 

One  of  the  systematic  and  more  objective  methods  of  assessing  the  quality 
of  the  programs  is  through  the  use  of  evaluation  research  techniques.  lo- 
using comparative  studies,  quasi-control  goups,  estimates  and  other  techniques, 
an  evaluator  can  choose  the  most  appropriate  method  or  combination  of  methods 
to  arrive  at  a  scientific,  objective,  and  comprehensive  assessment  of  a  pro- 
gram' s  effectiveness. 

This  two  part  bibliography  presents  citations  to  journal  and  monographic 
literature  on  the  broad  methodologies  of  evaluation  research  and  its  theories 
and  techniques  in  part  one,  and  in  part  two,  their  specific  application  to 
manpower  training  programs.  The  majority  of  the  citations  are  annotated  and 
have  been  chosen  from  the  computerized  bibliographic  data  base  created  and 
maintained  by  the  Library  Services  Division  of  the  Congressional  Research 
Service. 

Another  source  of  information  concerning  the  effectiveness  of  government 
sponsored  manpower  training  programs  are  the  reports  of  audits  conducted  and 
published  by  the  General  Accounting  Office.  The  primary  purpose  of  its  audits 
is  to  make  independent  examinations  for  the  Congress  of  the  ways  government 
agencies  are  discharging  their  financial  responsibilities.   In  its  audits 
the  General  Accounting  Cffice  makes  recommendations  to  effect  greater  economy 
and  efficiency  in  agency  operations  and  attempts  to  determine  whether  alternative 
approaches  might  better  accomplish  the  goals  of  the  program.  This  bibliography 
cites  several  of  the  reports  of  the  Comptroller  General;  others  may  be  located 
through  the  monthly  catalog  of  reports  issued  by  the  General  Accounting  Office. 

The  material  cited  is  arranged  alphabetically  by  main  entry  within  each 
of  the  two  sections.  Citations  to  items  in  the  classified  collection  of  the 
Library  of  Congress  are  provided  with  call  numbers,  items  designated  Law  are 
located  in  the  Law  Library  and  those  with  Newsp  are  found  in  the  Newspaper  and 
Current  Periodical  Room  of  the  Library  of  Congress.  All  journal  articles  are 
provided  with  the  call  number  for  the  bound  volumes  of  the  journal  regardless 
of  whether  the  issues  cited  have  been  bound  at  this  time. 
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CRS-2 


Evaluation  Research:  Theories  and  Techniques 

Buchanan,  Garth,  Pamela  Horst,  and  John  Scanlon.   Improving  Federal  evalua- 
tion planning.  Evaluation,  v.  1,  no.  2,  1973:  86-90. 

H62.A1E9,  v.  1 
Describes  "an  approach  to  federal  evaluation  planning  that  the  Urban 
Institute  has  been  developing  in  work  for  the  Department  of  Labor  and  the 
National  Institute  of  Mental  Health." 

Federal  evaluation  policy:  analyzing  the  effects  of  public  programs,  by  Joseph 
S.  Wholey  (and  others)  Washington,  Urban  Institute  £197(0  134-  p. 

HN59.F37 
Partial  contents. — Federal  evaluation — what  it  is  and  why  it  is 
needed. — Administration  of  an  evaluation  system. — Organizational  rela- 
tionships and  responsibilities. — Evaluation  resources. — Methodology. 
Bibliography:  p.  121-134. 

Garn,  Harvey  A. ,  and  Mancur  Olson.   Public  services  on  the  assembly  line. 
Evaluation,  v.  1,  no.  2,  1973:  36-42,  96.         H62.A1E9,  v.  1 

In  companion  articles,  the  authors  expand  on  evaluation  and  the 
care  that  must  be  exercised  when  measuring  public  performance. 

Goodwin,  Leonard.   Bridging  the  gap  between  social  research  and  public  policy: 
welfare,  a  case  in  point.  Washington,  Brookings  Institution,  1973.  114  p. 
(Brookings  Institution,  Washington,  D.C.  Reprint  266) 

Suggests  that  an  approach  to  social  problems  called  experimental 
social  research  can  help  social  scientists  recognize  social  events  as 
social  problems. 

Guttentag,  Marcia.   Subjectivity  and  its  use  in  evaluation  research. 
Evaluation,  v.  1,  no.  2,  1973:  60-65.  H62.A1E9,  v.  1 

Describes  a  model  for  program  evaluation  which  quantifies  decision- 
maker's values  on  the  belief  that  the  inherent  subjectivity  of  decision- 
making makes  classical  experimental  designs  not  very  useful. 

Lyden,  Fremont  James,  and  Lawrence  K.  Lee.   Evaluating  program  change. 
Social  work,  v.  18,  Mar.  1973:  87-94.  HV1.S644,  v.18 

"The  authors  describe  a  methodology  developed  and  used  effec- 
tively in  one  metropolitan  area  to  measure  program  change.  They 
identify  and  utilize  the  findings  to  show  how  greater  change  might  be 
effected  in  more  agencies." 

Meinert,  Roland  G.   Futures  forecasting.  Social  work,  v.  18,  Nov.  1973: 
48-51.  HV1.S644,  v.  18 

"Four  systematic  techniques  of  forecasting  the  future,  widely 
practiced  in  other  fields,  could  be  useful  in  social  planning.  The 
techniques  are  extrapolation,  Delphi  forecasting,  simulation,  and 
scenario  speculation." 

Shapley,  Deborah.   NSF:  engineers'  policy  group  urges  more  software  for 
RANN.   Science,  v.  178,  Dec.  22,  1972:  1270-1273.   Q1.S35,  v.  178 
"RANN  must  get  involved  in  the  nitty-gritty  of  institutional 
malfunctioning  and  reform  as  the  key  to  answering  national  needs." 
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CRS-3 

Symposium  on  policy  analysis.   Urban  affairs  quarterly,  v.  7,  June  1972: 
379-412.  HT101.U67,  v.  7 

Partial  contents. — Some  promising  approaches  to  policy  analysis: 
an  introduction,  by  J.  Brandl. — Microanalytie  simulation  modeling  for 
evaluation  of  public  policy,  by  H.  Guthrie. — The  use  of  experimentation 
in  policy  formulation  and  evaluation,  by  W.  Harrar  and  D.  Bawden. — 
The  capacity  of  social  science  organizations  to  perform  large-scale 
evaluative  research,  by  W.  Williams. 

Wegner,  Robert  E.  C.   Evaluating  the  evaluators.   Public  administration 
review,  v.  33,  Jan. -Feb.  1973:  80-87.  JK1.P85,  v.  33 

Reviews  A   recent  books  which  deal  with  evaluation  of  social  action 
programs;  books  are  by  Joseph  Keishaw,  Robert  Levine,  Alice  Rivlin  and 
Walter  Williams. 

Weiss,  Carol  H.   Abstracts  on  evaluative  research,  by  Carol  H.  Weiss  [and 

others,  n.pTJ  Model  Cities  Evaluation  Institute  [j  973?)   1  v.  (unpaged) 

Extensive  abstracts  of  articles  and  books  on  evaluation  in  a 
variety  of  program  areas  is  arranged  alphabetically  by  author;  subject/ 
author  index  is  provided. 


Between  the  cup  and  the  lip...  Evaluation,  v.  1,  no.  2,  1973: 

49-55.  H62.A1E9,  v.  1 

"That  much  evaluation  research  falls  short  of  the  expectations 
of  funders,  program  staff,  and  evaluators  themselves  is  due  less  to 
limitations  of  research  expertise  than  to  organizational  constraints 
on  the  researcher's  ability  to  apply  what  he  knows." 

Wholey,  Joseph  S.   What  can  we  actually  get  from  program  evaluation? 
Palicy  sciences,  v.  3,  Sept.  1972:  361-369.       H1.P7,  v.  3 

"Assesses  the  role  program  evaluation  can  play  in  assisting 
decisions  on  public  programs." 

Wilemon,  David.   Transferring  space-age  management  technology.  Conference 
Board  record,  v.  7,  Oct.  1970:  50>55.  HC101.C64,  v.  7 

"This  article  explores  the  characteristics  of  project  management 
to  appraise  its  transfer  potentialities  in  managing  society' s  complex 
problems,  first  examining.. .some  of  the  generalized  characteristics 
of  these  new  management  techniques." 
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CRS-4 
Evaluations  of  Manpower  Programs 

Alleviating  economic  distress;  evaluating  a  Federal  effort,  by  Raymond  H. 
Milkman  (and  others]   Lexington,  Mass.,  Lexington  Books  ft  972]  4.12  p. 

HC110.P63A7 

Barsby,  Steve  L.   Cost-benefit  analysis  and  manpower  programs.  Lexington, 
Mass.,  D.C.  Heath,  1972.  180  p.  HD5715.2.B36 

Concludes  that  vocational  rehabilitation  appears  to  yield  the 
highest  benefit-cost  ratios,  followed  by  vocational  education  programs 
in  vocational  high  schools,  out-of-school  institutional  retraining 
programs,  post  secondary  vocational  education,  Job  Corps,  and  dropout 
prevention  programs.  Barsby  adds  the  caveat  however,  that  comparisons 
of  manpower  programs  are  hindered  by  internal  deficiencies  in  the 
economic  theories  of  cost-benefit  analysis. 

The  economic  effectiveness  of  on-the-job  training:  the  experience 

of  the  Bureau  of  Indian  Affairs  in  Oklahoma:  comment.   Industrial 
and  labor  relations  review,  v.  24,  Jan.  1971:  265-258. 

HD4802.I53,  v.  24 
Boise  Cascade  Center  for  Community  Development.    Indian  economic  develop- 
ment; an  evaluation  of  EDA' s  Selected  Indian  Reservation  Program. 
Qjashington,  U.S.  Economic  Development  Administration}  1972.  82  p. 
(U.S.  Economic  Development  Administration.  EDA-72-59847) 

"Volume  I:  narrative"  E98.E2B64 

Bonis,  Michael  E. ,  and  William  R.  Task.  Measuring  the  impact  of  manpower 

programs;  a  primer.  Ann  Arbor,  Institute  of  Labor  and  Industrial  Rela- 
tions, University  of  Michigan-Wayne  State  University,  1970.  81  p. 
(Policy  papers  in  human  resources  and  industrial  relations,  no.  17) 

HD5724.B625 

Conference  on  the  Evaluation  of  the  Impact  of  Manpower  Programs,  Ohio 

State  University,  1971.   Evaluating  the  impact  of  manpower  programs; 
proceedings.  Edited  by  Michael  E.  Borus.  Lexington,  Mass.,  Lexington 
Books  ft  972]  280  p.  HD5724.C684  1971 

Bibliography:  p.  259-271 . 

Conly,  Sonia.   Success  and  failure  in  the  work  incentive  program.  Growth 
and  change,  v.  4,  Oct.  1973:  16-23. 

Presents  pilot  cost-benefit  study  of  the  WIN  program  for  welfare 
mothers  in  Florence,  South  Carolina. 

Craft,  James  A.  Public  service  jobs  and  transitional  employment:  an  analysis 
of  the  Vermont  Experimental  and  Demonstration  Project.  Pittsburgh, 
University  of  Pittsburgh  0974]  200  p. 

Garfinkel,  Irvin,  and  Larry  L.  Orr.   Welfare  policy  and  the  employment  rate 
of  the  AFDC  mothers.  National  tax  journal,  v.  27,  June  1974:  275-284. 

HJ2240.N3135,  v.  27 
"The  paper  analyzes  the  effect  of  economic  incentives  in  the  AFDC 
program  on  the  employment  of  AFDC  mothers.  Short  of  a  drastic  reduc- 
tion in  guarantees,  however,  manipulation  of  any  of  the  program  para- 
meters is  not  likely  to  induce  most  AFDC  mothers  to  work. 

Gold,  Stephen  F.   The  failure  of  the  Work  Incentive  (WIN)  Program.  Univer- 
sity of  Pennsylvania  law  review,  v.  119,  Jan.  1971:  485-501. 

Lav 
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Hanushek,  Eric  A.   Regional  differences  in  the  structure  of  earnings. 
Review  of  economics  and  statistics,  v.  55,  May  1973:  204.-213. 

HA1.R35,  v.  55 
"This  research  reveals  large  differences  across  labor  markets  in 
the  returns  to  human  capital  £and  educatioiQ  and  indicates  that  much  of 
the  observed  difference  in  regional  and  racial  earnings  results  from 
structural  differences  in  earnings  functions." 

Hardin,  Einar,  and  Michael  E.  Bonis.   Benefits  and  costs  of  MDTA-ARA  re- 
training.  Industrial  relations,  v.  11,  May  1972:  216-228. 

HD6951.I52,  v.  11 
Studying  the  manpower  training  programs  in  Michigan  under  the  Area 
Development  Act  (ADA)  and  the  Manpower  Development  and  Training  Act  (MDTA), 
the  authors  evaluate  the  impact  of  these  programs  on  the  national  pro- 
duct, the  disposal  income  of  trainees,  and  the  expenditures  and  receipts 
of  the  government. 


The  economic  benefits  and  costs  of  retraining.  Lexington,  Mass., 

Heath  Lexington  Books  C1971D  235  p.  HD571 5.3.M5H37  1971 

Economic  benefits  and  costs  of  retraining  courses  in  Michigan. 

Prepared  for  the  Manpower  Administration,  U.S.  Dept.  of  Labor.  East 
Lansing,  Mich.   ^School  of  Labor  and  Industrial  Relations,  Michigan 
State  University!  1%9.  428  p.  HD5715.3-.M5H37 

Holt,  Charles  C.   The  unemployment-inflation  dilemma:  a  manpower  solution. 
Washington,  Urban  Institute,  1971.  112  p.        HD572A.U6113 


Manpower  programs  to  reduce  inflation  and  unemployment,  by  Charles 

C.  Holt§nd  others]  Washington,  Urban  Institute,  1971. 

Lederer,  K.  M. ,  and  M.  P.  Bourgette.  A  critique  of  evaluation  studies  of 

manpower  policies  and  programs.  ^Edmonton,  Alberta,  Human  Resources 
Research  Council,  1972}  81  p.  Z7164.L1L34 

Bibliography:  p.  [28J-81. 

Levi tan,  Sar  A.  Antipoverty  work  and  training  efforts:  goals  and  reality. 
2d  ed.  $nn  Arbor,  Institute  of  Labor  and  Industrial  Relations)  1970. 
120  p.  HD6273.R7  1970 

A  joint  publication  of  the  Institute  of  Labor  and  Industrial 
Relations  (University  of  Michigan-Wayne  State  University)  and  the 
National  Manpower  Policy  Task  Force. 

Economic  opportunity  in  the  ghetto:  the  partnership  of  government 

and  business,  by  Sar  A.  Levi tan,  Garth  Mangum,  and  Robert  Taggert. 
Baltimore;  Johns  Hopkins  Press  C197Q)  84  p.      HC1 1 0 .  F63L395 

The  Emergency  Employment  Act;  an  interim  assessment.  Washington, 

U.S.  Govt.  Print.  Off.,  1972.  274  p. 

Prepared  for  the  Subcommittee  on  Employment,  Manpower,  and  Poverty 
of  the  Committee  on  Labor  and  Public  Welfare,  United  States  Senate. 

At  head  of  title:  92d  Congress,  2d  session.  Committee  print. 
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Levitan,  Sar  A.   Emergency  Employment  Act;  the  PEP  generation,  by  Sar  A. 
Levitan  and  Robert  Taggert.  Salt  Lake  City,  Utah,  Olympus  Publishing, 
d974.  257  p. 

Provides  selected  summaries  which  analyze  the  two  year  public 
employment  program  as  it  operated  in  various  cities. 


Evaluation  of  the  first  18  months  of  the  public  employment  program, 

by  Sar  A.  Levitan  and  Robert  Taggert.  Prepared  for  the  Subcommittee  on 
Employment,  Poverty,  and  Migrant  Labor  of  the  Committee  on  Labor  and 
Public  Welfare,  United  States  Senate.  Washington,  U.S.  Govt.  Print. 
Off.,  1973.  44  p.  HD5724.L4226 

At  head  of  title:  93d  Congress,  1st  session.  Committee  print. 

Social  experimentation  and  manpower  policy:  the  rhetoric  and  the 

reality,  by  Sar  A.  Levitan  and  Robert  Taggert.  Baltimore,  Johns  Hopkins 
Press  p97?]  111  p.  HD5724.H23 

Work  and  welfare  go  together  (by)  Sar  A.  Levitan,  Martin  Rein  [and] 

David  Marwick.   Baltimore,  Johns  Hopkins  Press  £1972}   143  p. 

HV95.L55 

Mangum,  Garth  L. ,  and  John  Walsh.   A  decade  of  manpower  development  and 
training.  Salt  Lake  City,  Utah,  Olympus  Publishing,  1973.  151  p. 

HD5715.2.M35 
Authors  conclude  that  MDTA  has  been  worth  the  cost  and  the  effort. 
The  evidence  they  found  supports  its  effectiveness  as  a  device  to  increase 
the  employment  of  enrollees  and  it  has  opened  new  occupations  and  avenues 
of  employment  to  both  trainees  and  staff.  They  find,  however,  that  MDTA 
does  need  to  focus  on  broader  concepts  of  employability,  particularly 
prestigious  institutions  taking  special  care  to  see  that  the  disadvan- 
taged are  served. 

Minnesota.  University.  Center  for  Youth  Development  and  Research.   An  in- 
ventory and  assessment  of  youth  development  centers  at  colleges  and 
universities  in  the  United  States.  Qlinneapolis ,  1971]  183  1. 

HQ796.M534 

Sewell,  David  0.   Training  the  poor;  a  benefit-cost  analysis  of  manpower 

programs  in  the  U.S.  antipoverty  program.  Kingston,  Ontario,  Industrial 
Relations  Center,  Queens's  University,  1971.  153  p.   (Queen's  University. 
Industrial  Relations  Centre.  Research  series,  no.  12) 

HD5724.S47 

Somers,  Gerald  G.,  and  Ernst  W.  Stromsdorfer.   A  cost-effectiveness  analysis 
of  in-school  and  summer  Neighborhood  Youth  Corps:  a  nationwide  evaluation. 
Journal  of  human  resources,  v.  47 ,  fall  1972:  446-459. 

HD5701.J6,  v.  47 
Study  concluded  that  the  "composite  of  in-school  and  summer  NYC 
programs  included  in  this  survey  yields  significant  monetary  benefits, 
due  mainly  to  an  effect  on  increasing  labor  force  participation.   On  this 
basis,  the  program  warrants  continued  and  perhaps  increased  funding  as 
an  effective  social  program.  The  summer  NYC  program,  alone  does  not 
warrant  funding  on  the  basis  of  economic  efficiency,  but  funding  may  well 
be  justified  on  social  grounds." 
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Stromsdorfer,  Ernest  W.   Review  and  synthesis  of  cost-effectiveness  studies 
of  vocational  and  technical  education.  Monticello,  111.,  1973.  A2   p. 
(Council  of  Planning  Librarians.  Exchange  bibliography  362) 

U.S.  Congress.  Joint  Economic  Committee.  Subcommittee  on  Fiscal  Policy. 
Studies  in  public  welfare;  paper  no.  3.  Washington,  U.S.  Govt.  Print. 
Off.,  1972.  70  p. 

Paper  no.  3:  The  effectiveness  of  manpower  training  programs:  a 
review  of  research  on  the  impact  of  the  poor. 

U.S.  Congress.  Senate.   Committee  on  Labor  and  Public  Welfare.  Subcommit- 
tee on  Employment,  Poverty,  and  Migratory  Labor.   Case  studies  of  the 
Emergency  Employment  Act  in  operation.  Washington,  U.S.  Govt.  Print. 
Off.,  1973.  U10  p. 

At  head  of  title:  93d  Congress,  1st  session.   Committee  print. 

U.S.  Dept.  of  Labor.  Manpower  Administration.   Manpower  research  and  deve- 
lopment projects.  (^Washington  1973]  330  p.      HD5723.A4.U 

Reports  on  4-55  research  and  development  projects  covering  these 
subjects:  programs  for  the  use  and  development  of  manpower;  special 
target  groups;  manpower  program  planning  and  administration;  the  labor 
market;  and  manpower,  economic,  and  social  policies. 

U.S.  Economic  Development  Administration.   Developing  methodologies  for 

evaluating  the  impact  of  EDA  programs.  CWashington]  1972.  1  v.   (various 
pagings)  HC110.P63A5U  1972 

U.S.  General  Accounting  Office.   Assessment  of  operations  and  management  of 
Opportunities  Industrialization  Centers,  Department  of  Labor,  Department 
of  Health,  Education,  and  Welfare,  Office  of  Economic  Opportunity;  report 
to  the  Congress  by  the  Comptroller  General  of  the  United  States. 
CWashington]  1972.  99  p.  HD5715.2.A554. 

"B-U6879,  Apr.  20,  1972" 

Efforts  to  employ  disadvantaged  persons  in  the  Federal  Government, 

Civil  Service  Commission,  Department  of  Labor;  report  to  the  Congress  by 
the  Comptroller  General  of  the  United  States.  CWashington)  1972.  61  p. 
"B-163922,  April  17,  1972"  JK765.A529 

Evaluation  of  results  and  administration  of  the  Job  Opportunities  in 

the  Business  Sector  (JOBS)  Program  in  five  cities;  report  to  the  Congress 
(pri  the]  Department  of  Labor  by  the  Comptroller  General  of  the  United 
States.   Washington,  1971^  111  p.  HD  5715.2.A555 

"B-163922,  Mar.  2U,   1971" 

Federal  manpower  training  programs — GAO  conclusions  and  observations, 

Department  of  Labor,  Department  of  Health,  Education,  and  Welfare,  Office 
of  Economic  Opportunity;  report  to  the  Committee  on  Appropriations, 
United  States  Senate  by  the  Comptroller  General  of  the  United  States. 
[Washington]  1972.  58  p.  HD5715.2.A556 

"B-1 46879,  Feb.  17,  1972" 
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U.S.  General  Accounting  Office.   Opportunities  to  increase  effectiveness  of 
long-term,  full-time  training  program  for  civilian  employees,  Department 
of  Defense;  report  to  the  Congress  by  the  Comptroller  General  of  the 
United  States.   QfeshingtonJ  1972.  45  p.         UB193.U54  1972 
"B-70896,  June  30,  1972" 

Problems  in  accomplishing  objectives  of  the  Work  Incentive  Program 

(WIN);  report  to  the  Congress  [on  the]  Department  of  Labor  [and  the] 
Department  of  Health,  Education,  and  Welfare,  by  the  Comptroller  Gen- 
eral of  the  United  States.  (^Washington,  1 971J  55  p. 

HD5715.2.A56 
"B-164031  (3)  Sept.  24,  1971" 

Restructured  Neighborhood  Youth  Corps  out-of-school  program  in  urban 

areas.   Department  of  Labor;  report  to  the  Congress  by  the  Comptroller 
General  of  the  United  States.  [^Washington]  1974.  41  p. 

"B-130515,  Apr.  2,  1974"  HD6273.U52  1974 

Wilson,  John  Oliver.   Social  experimentation  and  public-policy  analysis. 
Public  policy,  v.  22,  winter  1974:  15-37.         JA51.P8,  v.  22 

Examines  the  strengths  and  weaknesses  of  social  experimentation  using 
examples  such  as  evaluation  of  the  New  Jersey  Guaranteed  Work  Incentive 
Program. 

Wisconsin.   University.   Institute  for  Research  on  Poverty.   Final  report  of 
the  New  Jersey  graduated  work  incentive  experiment,  by  Institute  for 
Research  on  Poverty,  University  of  Wisconsin,  Madison,  and  Mathematica. 
Madison  p974J  5  v. 

Partial  contents. — Central  labor  supply  response. — Studies  relating 
to  the  validity  and  generalizability  of  the  results. — Response  with  re- 
spect to  expenditure,  health,  and  social  behavior. — Operations,  surveys, 
and  administration. 


The  New  Jersey  graduated  work  incentive  experiment;  summary  report 

by  Institute  for  Research  on  Poverty,  University  of  Wisconsin,  Madison, 
and  Mathematica,  inc.  flashing-ton]  U.S.  Office  of  Economic  Opportunity, 
1973.  1  v.  (various  pagings) 

Wright,  George  N.,  and  Kenneth  W.  Reagles.   The  economic  impact  of  an  ex- 
panded program  of  vocational  rehabilitation.  Madison,  University  of 
Wisconsin,  Regional  Rehabilitation  Research  Institute,  1971.  69  p. 
(Wisconsin  studies  in  vocational  rehabilitation.  Monograph  15,  series  2) 

HD7265.U6W578 
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Evaluation  Research  in  Public  Administration: 
Selected  References,  1967-1974. 

The  continuously  active  process  of  running  the  business  of  government 
has  grown  more  expensive,  more  diffuse,  and  more  complex.   Traditionally, 
public  administration  has  been  concerned  with  carrying  out  the  law  effectively, 
efficiently,  and  economically  through  organization  and  management.  More 
recently  it  has  concerned  itself  with  interpreting  the  law  as  well.  Public 
administration,  whether  an  art  or  a  science,  is  in  a  period  of  intense  study 
and  reflection.  One  of  the  systematic  and  more  objective  methods  of  assessing 
the  quality  of  its  efforts  is  through  evaluation  research. 

The  broad  purpose  of  attaining  a  scientific,  objective,  and  compre- 
hensive assessment  of  the  effectiveness  of  a  program  gives  evaluation  research 
a  common  bond  with  other  social  science  research.  However,  the  conditions 
under  which  evaluation  research  must  operate  provide  it  with  a  character 
distinct  from  other  social  science  research.   The  evaluator  frequently 
cannot  manipulate  the  program  or  control  the  testing  conditions  because 
of  the  interactive  environment  of  an  administrative  process  and/or  its 
multiple  goals.  But  through  the  use  of  comparative  studies,  quasi-control 
groups,  estimates  or  other  techniques,  an  evaluator  can  choose  the  most 
appropriate  method  or  combination  of  methods  to  provide  a  scientific  assess- 
ment. While  it  is  immediately  apparent  that  each  evaluation  must  be 
tailored  to  the  particular  needs  of  the  programs  being  studied,  there 
are  broad  methodologies  common  to  most  evaluation  research. 

The  following  two  part  bibliography  presents  citations  to  journal  and 
monographic  literature  on  the  broad  methodologies  of  evaluation  research 
and  its  theories  and  techniques  in  part  one,  and  in  part  two,  their  specific 
application  to  public  administration  and  selected  public  programs.  The 
majority  of  the  citations  are  annotated  and  have  been  chosen  from  the 
computerized  bibliographic  data  base  created  and  maintained  by  the  Library 
Services  Division  of  the  Congressional  Research  Service. 

Another  source  of  information  concerning  the  effectiveness  of  government 
agencies  and  their  programs  are  the  reports  of  audits  conducted  and 
published  by  the  General  Accounting  Office.  The  primary  purpose  of  its 
audits  is  to  make  independent  examinations  for  the  Congress  of  the  ways 
government  agencies  are  discharging  their  financial  responsibilities. 
In  its  audits  the  General  Accounting  Office  makes  recommendations  to  effect 
greater  economy  and  efficiency  in  agency  operations  and  attempts  to  deter- 
mine whether  alternative  approaches  might  better  accomplish  the  goals  of  the 
program.  This  bibliography  cites  only  a  few  of  the  reports  of  the  Comptroller 
General;  others  may  be  located  in  the  monthly  catalog  of  reports  issued  by 
the  General  Accounting  Office.      • 

The  material  cited  is  arranged  alphabetically  by  main  entry  within  each 
of  the  two  sections.  Citations  to  items  in  the  classified  collection  of  the 
Library  of  Congress  are  provided  with  call  numbers;  items  designated  Law  are 
located  in  the  Law  Library  and  those  with  Newsp  are  found  in  the  Newspaper 
and  Current  Periodical  Room  of  the  Library  of  Congress.   All  journal  articles 
are  provided  with  the  call  number  for  the  bound  volumes  of  the  journal 
regardless  of  whether  the  issues  cited  have  been  bound  at  this  time. 
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Biderman,    Albert  D. ,   and  Laure  M.   Sharp.       The  competitive  evaluation  research 
industry.     Washington,    Bureau  of  Social  Science  Research,   1972.     75  p. 
Examines  the  social  program  evaluation  research  "industry"  which 
has  recently  appeared  in  the  U.S.   and  the  competitive  aspects  of  that 
industry  (in  bidding  for  contracts,    etc.). 

Buchanan,   Garth,   Pamela  Horst,   and  John  Scanlon.        Improving  Federal  evaluation 
planning.      Evaluation,   v.   1,   no.   2,   1973:    86-90.  H62.A1E9,   v.  1 

Describes  an  approach  to  Federal  evaluation  planning  that  the  Urban 
Institute  has  been  developing  in  work  for  the  Department  of  Labor  and  the 
National  Institute  of  Mental  Health. 

Caro,   Francis  G.,    comp.        Readings  in  evaluation  research.      New  York,   Russell 
Sage  Foundation   [1971]      418  p.  H62.C3425 

Davis,    Howard  R.        Four  ways  to  goal  attainment;   an  overview.      Evaluation,   v.   1, 
no.   2,   1973:   43-48.  H62.A1E9,    v.   1 

Outlines  four  different  methods  of  goal  attainment  evaluation. 

Federal  evaluation  policy:    analyzing  the  effects  of  public  programs,   by  Joseph 
S.   Wholey  [and  others]     Washington,   Urban  Institute   [1970]      134  p. 

HN59.F37 
Partial  contents. — Federal  evaluation — what  it  is  and  why  it  is 
needed. — Administration  of  an  evaluation  system. — Organizational  relation- 
ships and  responsibilities. — Evaluation  resources. — Methodology. 
Bibliography:   p.   121-134. 

Goodwin,   Leonard.        Bridging  the  gap  between   social  research  and  public  policy: 
welfare,   a  case  in  point.     Washington,   Brookings  Institution,   1973.     114  p. 
(Brookings  Institution,   Washington,   D.C.      Reprint  266) 

Suggests  that  an  approach  to  social  problems  called  experimental 
social  research  can  help  social  scientists  recognize  social  events  as 
social  problems. 

Guttentag,   Marcia.        Subjectivity  and  its  use  in  evaluation  research. 

Evaluation,    v.   1,   no.    2,   1973:    60-65.  H62.A1E9,  v.   1 

Describes  a  model  for  program  evaluation  which  quantifies  a 
decision-maker's  values  in  the  belief  that  the  inherent  subjectivity 
of  decision-making  makes  classical  experimental  designs  not  very  useful. 

Lyden,   Fremont  James,   and  Lawren<*e  K.   Lee.        Evaluating  program  change. 
Social  work,   v.   18,   Mar.   1973:   87-94.  HV1.S644,   v.  18 

"The  authors  describe  a  methodology  developed  and  used  effectively 
in  one  metropolitan  area  to  measure  program  change.     They  identify  and 
utilize  the  findings  to   show  how  greater  change  might  be  effected  in 
more  agencies." 

Meinert,   Roland  G.        Futures  forecasting.      Social  work,   v.   18,    Nov.   1973: 
48-51.  HV1.S644,  v.  18 

"Four  systematic  techniques  of  forecasting  the  future,   widely 
practiced  in  other  fields,    could  be  useful  in  social  planning.     The 
techniques  are  extrapolation,   Delphi  forecasting,    simulation,   and 
scenario   speculation." 
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Rossi,  Peter  Henry,  and  Walter  Williams,  eds.   Evaluating  social  action 

programs;  theory,  practice,  and  politics.  New  York,  Seminar  Press,  1972. 
326  p.  H62.R6U 

Shapley,  Deborah.   NSF:  engineers'  policy  group  urges  more  software  for  RANN. 
Science,  v.  178,  Dec.  22,  1972:  1270-1273.         Q1.S35,  v.  178 

"RANN  must  get  involved  in  the  nitty-gritty  of  institutional  mal- 
functioning and  reform  as  the  key  to  answering  national  needs." 

Suchman,  Edward  Allen.   Evaluation  research;  principles  and  practice  in 
public  service  &  social  action  programs.  New  York,  Russell  Sage 
Foundation,  1967.  186  p.  H61.S888 

Symposium  on  policy  analysis.  Urban  affairs  quarterly,  v.  7,  June  1972: 
379-iU2.  HT101.U67,  v.  7 

Contents. — Some  promising  approaches  to  policy  analysis:  an 
introduction,  by  J.  Brandl. — An  advocate's  process  outline  for  policy 
analysis:  the  case  of  welfare  reform,  by  J.  Lyday. — Microanalytic 
simulation  modeling  for  evaluation  of  public  policy,  by  H.  Guthrie. — 
The  use  of  experimentation  in  policy  formulation  and  evaluation,  by 
W.  Harrar  and  D.  Bawden. — The  capacity  of  social  science  organizations 
to  perform  large-scale  evaluative  research,  by  W.  Williams. 

Wegner,  Robert  E.  C.   Evaluating  the  evaluators.  Public  administration 
review,  v.  33,  Jan.-Feb.  1973:  80-87.  JK1.P85,  v.  33 

Reviews  four  recent  books  by  Joseph  Keishaw,  Robert  Levine, 
Alice  Rivlin  and  Walter  Williams  which  deal  with  evaluation  of  social 
action  programs. 

Weiss,  Carol  H.   Abstracts  on  evaluative  research,  by  Carol  H.  Weiss 

[and  others,  n.pj  Model  Cities  Evaluation  Institute  D.973?)   1  v. 
(unpaged). 

Extensive  abstracts  of  articles  and  books  on  evaluation  in 
a  variety  of  program  areas  are  arranged  alphabetically  by  author 
and  a  subject/ author  index  is  provided. 


-  Between  the  cup  and  the  lip...  Evaluation,  v.  1,  no.  2,  1973: 
49-55.  H62.A1E9,  v.  1 

"That  much  evaluation  research  falls  short  of  the  expectations 
of  funders,  program  staff,  and  evaluators  themselves  is  due  less  to 
limitations  of  research  expertise  than  to  organizational  constraints 
on  the  researcher's  ability  to  apply  what  he  knows." 

-  Evaluating  action  programs;  readings  in  social  action  and  education, 
(compiled  by)  Carol  H.  Weiss.  Boston,  Allyn  and  Bacon  C-972J   365  p. 

H62.W396 
Partial  contents. — A  critique  of  evaluative  research,  by  E.  A. 
Suchman. — Memorandum  on  program  evaluation,  by  H.  W.  Reicken. — 
Evaluation  theory  development,  by  M.  C.  Alkin. — Booby  traps  and 
pitfalls  in  the  evaluation  of  social  action  programs,  by  P.  H.  Rossi. 

-  Evaluation  research:  methods  for  assessing  program  effectiveness. 
Englewood  Cliffs,  N.J.,  Prentice-Hall  [1972J   160  p. 

H62.W3962 
Bibliography:  p.  129-154. 
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An  introduction  to  sample  surveys  for  government  managers  Qsy]  Carol 

H.  Weiss  [and}  Harry  P.  Hatry.   Washington,  Urban  Institute,  1971. 
A8   p.  HN29.W4 

Wholey,  Joseph  S.   What  can  we  actually  get  from  program  evaluation?   Policy 
science,  v.  3,  Sept.  1972:  361-369. 

"Assesses  the  role  program  evaluation  can  play  in  assisting 
decisions  on  public  programs." 

Wilemon,  David.   Transferring  space-age  management  technology.   Conference 
Board  record,  v.  7,  Oct.  1970:  50-55.  HC101.C64.,  v.  7 

"This  article  explores  the  characteristics  of  project  management 
to  appraise  its  transfer  potentialities  in  managing  society's  complex 
problems,  first  examining. . .some  of  the  generalized  characteristics 
of  these  new  management  techniques." 

Williams,  Walter.   The  capacity  of  social  science  organizations  to  perform 
large-scale  evaluative  research.   Seattle  Qnstitute  °f  Governmental 
Research,  University  of  Washington)  1971.  4-7  p.   (Public  policy  paper, 
no.  2)  JA37.P9  no. 2 
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Anthony,  Robert.   Closing  the  loop  between  planning  and  performance.   Public 
administration  review,  v.  31,   May- June  1971:  388-393. 

JK1.P85,  v.  31 

Banfield,  Edward  C.   12  sensible — but  unpopular — steps  toward  a  better  city. 
Chicago  tribune  magazine,  Apr.  19,  1970:  30-33,  UU,   tf>,   49-54,  59-60. 

Newsp 

Bell,  Daniel.   On  meritocracy  and  equality.   Public  interest,  no.  29, 
fall  1972:  29-68.  H1.P86,  no.  29 

Benefit-cost  analyses  of  Federal  programs;  a  compendium  of  papers.   Submitted 
to  the  Subcommittee  on  Priorities  and  Economy  in  Government  of  the 
Joint  Economic  Committee,  Congress  of  the  United  States.   Washington, 
U.S.  Govt.  Print.  Off.,  1973.   274  p.  HJ7539.B45 

At  head  of  title:  92d  Congress,  2d  session.   Joint  committee  print. 

Partial  contents. — Report  on  a  survey  of  Federal  program  evaluation 
practices. — Profiles  of  analytical  studies. — An  economic  perspective 
on  the  small  watersheds  program. — Efficiency  and  equity  effects  in  the 
benefits  from  the  Federal  housing  program  in  1965. — Econometric  analysis 
of  the  Medicare  program. — The  in-school  and  summer  Neighborhood  Youth 
Corps:  a  nationwide  evaluation  of  an  investment  in  disadvantaged  high 
school  youth. — Benefits  and  costs  of  manpower  training  programs:  a 
synthesis  of  previous  studies  with  reservations  and  recommendations. 

Brooks,  Michael.   Social  planning  and  city  planning.   Chicago,  American 
Society  of  Planning  Officials,  cl97tf]   61  p.       HT167.B7 

Contents. — Social  planning  in  the  United  States. — City  planning 
and  social  problems. — A  critical  look  at  advocacy  planning. — Toward 
a  more  effective  social  planning  process. — Bibliography. 

Chapman,  William.   Scholars  examine  value  of  social  scientists  in  public 
policy.   Washington  post,  Feb.  18,  1973,  p.  A2.     Newsp 

Discusses  the  debate  about  the  role  of  social  scientists  in 
public  policy  involving  Adam  Yarmclinsky,  James  Q.  Wilson  and  William 
M.  Capron. 

Committee  for  Economic  Development.   Improving  Federal  program  performance; 
a  statement  on  national  policy  by  the  Research  and  Policy  Committee  of 
the  Committee  for  Economic  Development,  September,  1971.  C^ew  York, 
19717  86  p.  HJ7539.C65 

Cox,  Bill.   Application  of  marketing  concepts  to  the  analysis  of  a  govern- 
ment program  or  agency.   GAO  review,  spring  1971:  60-67. 

HJ9701.G3 
The  GAO,  in  analyzing  management  of  government  programs,  has 
emphasized  only  failure  to  produce  or  distribute  needed  services. 
Author  points  out  that  "the  underlying  causes  for  varying  degrees  of 
program  ineffectiveness  can  be  found  in  three  other  areas:  defined 
need  does  not  exist,  unrealistic  allocation  of  resources,  or  an  im- 
properly defined  target  market." 
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Gam,  Harvey  A.,  and  Mancur  Olson.   Public  services  on  the  assembly  line. 
Evaluation,  v.  1,  no.  2,  1973:  36-42,  96.  H62.A1E9,  v.  1 

In  companion  articles,  the  authors  expand  on  evaluation  and  the 
care  that  must  be  exercised  when  measuring  public  performance. 

Gonzales,  Jack,  and  John  Rothchild.  The  Shriver  prescription:  how  the 
government  can  find  out  what  it's  doing.  Washington  monthly,  v.  L,, 
Nov.  1972:  32-40.  E838.W37,  v.  4 

Discusses  the  "antibureaucratic"  role  Shriver  took  when  running 
0E0  and  the  Peace  Corps. 

The  Great  society:  lessons  for  the  future.   Public  interest,  no.  34-,  winter 
1974:  whole  issue.  H1.P86,  no.  34 

Partial  contents. — Social  intervention  in  a  democracy,  by  L. 
Liebman. — Economic  policy  and  unemployment  in  the  1960's,  by  E.  Phelps. — 
Reform  follows  reality:  the  growth  of  welfare,  by  G.  Steiner. — What 
does  it  do  for  the  poor?  A  new  test  for  national  policy,  by  R.  Lampman.- 
-The  uses  and  limits  of  manpower  policy,  by  L.  Ulman. — Major  public 
initiatives  in  health  care,  by  H.  Klarman. — Economic  developments  in 
the  Black  community,  by  A.  Brimier. — The  Federal  role  in  education,  by 
R.  Tyler. — Some  lessons  of  the  1960's,  by  E.  Ginzberg  and  R.  Solow. 

Gustafson,  George  A.   Critical  issues  in  governmental  accounting.   Federal 
accountant,  v.  22,  Mar.  1973:  10-23.  HJ9801.F4,  v.  22 

Sets  forth  critical  current  issues  in  the  areas  of  financial,  per- 
formance, compliance,  social,  and  intergovernmental  accountability  on 
the  Federal,  state,  and  local  government  levels. 

Hargrove,  Erwin  C.   Professional  styles  in  government:  touchstones  for  the 
new  policy  scientist.  Policy  sciences,  v.  2,  summer  1971 :  229-246. 

H1.P7,  v.  2 
"The  article  reports  interviews  with  urban  planners,  lawyers, 
economists  and  political  scientists  about  their  experience  in  govern- 
ment.  Propositions  are  developed  about  characteristic  modes  of  thought 
and  problem-solving  styles  of  each  profession  and  conclusions  drawn 
about  the  relative  'effectiveness'  of  each  set  of  professional  skills 
in  a  policymaking  process.   It  is  then  suggested  that  the  new  policy 
scientist  profession  should  develop  some  of  the  positive  skills  of  the 
other  fields  in  order  to  enhance  effectiveness." 

Ha try,  Harry  P.   Criteria  for  evaluation  in  planning  state  and  local  programs; 
a  study  submitted  by  the  Subcommittee  on  Intergovernmental  Relations 
(pursuant  to  S.  Res.  55,  90th  Cong.)  to  the  Committee  on  Governmental 
Operations,  United  States  Senate.   Washington,  U.S.  Govt.  Print.  Off., 
1967.  42  p.  JF1351.H28 

At  head  of  title:  90th  Congress,  1st  session.   Committee  print. 

Improving  productivity  and  productivity  measurement  in  local  govern- 
ment [by]  Harry  P.  Hatry  and  Donald  M.  Fisk.  Prepared  for  the  National 
Commission- on  Productivity.   [Washington]  1971  [i.e.   1<^7^}      7^  P« 

JS36J.H37 
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Practical  program  evaluation  for  state  and  local  government  officials, 

bit  ■  E .  Batty,  Richard  E.  Winnie,  and  Donald  M.  Fisk.  Q/ashington) 

- 1  tute  £1  <773}     134.  p .  5 .  H33 

Partial  contents. — Scope  and  current  practices. — What's  the  program 
all  about?   Identification  of  program  objectives,  evaluation  criteria, 
and  clientele  groups. — Case  study  of  evaluation  in  action:  operation 
clean  sweep— a  special  city  cleanup. — Putting  it  all  together:  institu- 
tional issue;. 

p.  127-134. 

LeGates,  Richard  1.   Can  the  Federal  welfare  bureaucracies  control  their 
programs:  the  case  of  HUD  and  urban  renewal.   Urban  lawyer,  v.  5, 
spring  1973:  228-2-:. 3.  Law 

:es  "on  the  Urban  Renewal  program  and,  by  close  examination  of 
the  internal  workings  of  the  HUD  bureaucracy  that  administers  Urban 
Renewal,  seek£pj  to  explain  why  it  is  that  HUD — and  by  implication 
other  new  social  welfare  bureaucracies — cannot  control  their  programs." 

Lynn,  Laurence  E. ,  Jr.   A  Federal  evaluation  office?  Evaluation,  v.  1,  no.  2, 
1973:  56-59,  92,  96.  H62.A1E9,  v.  1 

Former  HEW  Assistant  Secretary  for  Planning  and  Evaluation  proposes 
extablishment  of  a  Federal  Program  Evaluation  Office  or  centering  all 
departmental  evaluation  functions  in  the  evaluation  office  of  each 
Secretary. 

Making  Federal  programs  work  better;  highlights  of  Systematic  thinking  for 
social  action,  by  Alice  M.  Rivlin.  Washington,  Brookings  Institution 
[c19713  9  p.   (Brookings  research  report  112)     H62.B714.  no.  112 

Marcuse,  Peter.   Social  indicators  and  housing  policy.  Urban  affairs 
quarterly,  v.  7,  Dec.  1971:  193-217.  HT101.U67,  v.  7 

"Existing  national  housing  policy  is  pursaing  unsatisfactory 
goals  and  relying  on  unsatisfactory  indicators  to  signal  its  achievements 
in  doing  so,  but  better  goal  formulations  are  hard  to  come  by." 

Morehouse,  Thomas  A.   The  problem  of  measuring  the  impacts  o'f  social-action 
programs.   [Fairbanks,  Institute  of  Social,  Economic  and  Government 
Research,  University  of  Alaska,  1972]   26  p.       H62.5.U5M67 
Bibliography:  p.  24.-2-,. 

Program  evaluatian:  social  research  versus  public  policy.  Public 

administration  review,  v.  32,  Nov.-Dec.  1972:  868-874-. 

JK1.P85,  v.  32 

Rivlin,  Alice  M.  '  Systematic  thinking  for  social  action.  Washington, 
Brookings  Institution  Ql97f]   150  p.  (H.  Rowan  lectures  in  svstems 
science)  H62.R56  * 

Staats,  Elmer  E.  Evaluating  the  effectiveness  of  Federal  social  programs. 
GAO  review,  fall  1973:  1-7.  HJ9701.G3 

"Some  of  the  difficulties  of  evaluating  social  programs  are  dis- 
cussed in  this  article  which  is  based  on  a  speech  delivered  by  the 
Comptroller  General  before  the  National  Capital  Area  chapter  of  the 
American  Society  of  Public  Administration  on  January  17,  1973." 
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Stober,  William  J.,  and  Laurence  H.  Falk.   Evaluating  investment  decisions 

of  state  and  local  governments.   Growth  and  change,  v.  1,  Oct.  1970:  38-4.2. 

Symposium  on  policy  analysis.  Urban  affairs  quarterly,  v.  7,  June  1972: 
379-412.  HT101.U67,  v.  7 

Partial  contents. — Some  promising  approaches  to  policy  analysis: 
an  introduction,  by  J.  Brandl. — Microanalytic  simulation  modeling  for 
evaluation  of  public  policy,  by  H.  Guthrie. —The  use  of  experimentation 
in  policy  formulation  and  evaluation,  by  W.  Harrar  and  D.  Bawden. — 
The  capacity  of  social  science  organizations  to  perform  large-scale 
evaluative  research,  by  W.  Williams. 

Systems  analysis  aids  in  urban  problem  areas.  NBS  technical  news  bulletin, 
v.55,  Dec.  1971:  296-297,  302.  QC100.U5753,  v.  55 

U.S.   Congress.   Joint  Economic  Committee.   Subcommittee  on  Economy  in 
Government.   Economic  analysis  and  the  efficiency  of  government. 
Hearings,  91st  Congress,  1st  and  2nd  sessions,  92d  Congress,  1st  session. 
Washington,  U.S.  Govt.  Print.  Off.,  1969-1971.  6  pts.  in  5  v.   (1308  p.) 

KF25.E243  1969a 

Hearings  held  Aug.  12,  1969  -  July  19,  1971. 

Parts  1-3  provide  a  general  overview  of  government  efficiency, 
while  the  last  three  parts  cover  specific  topics:  Part  4-Supersonic 
Transport  Development;  Part  5-Federal  Transportation  Policy;  Part  6- 
Economic  Incentives  to  Control  Pollution. 

U.S.  Congress.   Senate.   Committee  on  Labor  and  Public  Welfare.   Special 
Subcommittee  on  Evaluation  and  Planning  of  Social  Programs.   Full 
Opportunity  Act.  Hearings,  91st  Congress,  2d  session,  on  S.  5. 
Washington,  U.S.  Govt.  Print.  Off.,  1970.  4-39  p.   KF26.L386  1970 
Hearings  held  July  7  -  Dec.  18,  1969  and  Mar.  13,  1970. 

U.S.  Congress.   Senate.   Select  Committee  on  Nutrition  and  Human  Needs. 
Promises  to  keep:  housing  need  and  Federal  failure  in  rural  America. 
Rev.   Apr.  1972.  Washington,  U.S.  Govt.  Print.  Off.,  1972.  4-3  p. 

HD7289.U6A536  1972 
At  head  of  title:  92d  Congress,  2d  session.   Committee  print. 

U.S.   Dept.  of  Housing  and  "Jrban  Development.   The  Highland  Heights  experiment? 
a  final  report.  [Washington,  1973]   108  p. 

The  experiment  deals  with  the  action  goals  relating  to  the  screen- 
ing process  and  the  stimulation  of  services  for  residents  of  the  HUD 
sponsored  low  income  Fall  River  Highlands  Heights  Apartments  for  the 
physically  impaired  and  elderly.  The  report  deals  with  the  findings  of 
this  short-term  research  effort. 

U.S.   General  Accounting  Office.   ^Legislative  references  to  evaluation,  1967- 
1972.   Washington]  1972.  62  1. 

"B-161740,  Aug.  14-,  1973" 

"This  report  includes  a  review  of  evaluation  language  contained 
both  in  legislation  enacted  between  January  1967  and  July  1972  and  in 
bills  passed  by  either  or  both  Houses  of  Congress  from  January  1972 
through  July  1972." 
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program  evaluation:  legislative  language  and  a  user's  guide  to 

-elected  sources.   [Washington]  1973.   53  p. 

"Part  I  is  a  user's  guide  to  major  sources  of  specialized  studies, 
including  evaluation  studies,  which  are  available  outside  of  the  various 
departments  and  agencies....   Part  II  is  a  summary  of  evaluation  language 
contained  both  in  legislation  enacted  between  January  1967  and  July  1972 
and  in  bills  passed  by  either  or  both  Houses  of  the  Congress  from  January 
ougta  July  1972." 

an  Institute;  research,  1968-1971.  Operations  research,  v.  20,  May- 
June  1972:  516-557.  Q175.063,  v.  20 

Iption  of  the  first  three  years  of  research  at  the  Urban 
Institute,  a  nonprofit,  nongovernmental  research  organization  empha- 
sizing the  economic  and  social  aspects  of  urban  problems. 

********************* 

The  following  entries  were  received  too  late  to  be  included  with  the 
alphabetized  section  of  evaluation  studies.  These  citations  have  been 
chosen  from  the  Symposium  on  Program  Evaluation  which  appears  in  the  July- 
August  1974.  issue  of  Public  Administration  Review. 

Horst,  Pamela,  and  others.   program  management  and  the  Federal  evaluator. 
B  administration  review,  v.  34,  July- Aug.  1974:  300-308. 

JK1.P85,  v.  34 
Cites  three  conditions  which  render  a  program  "not  evaluable":  lack 
of  definition,  lack  of  clear  logic  and  lack  of  management.  Refusing  to 
perform  an  evaluation  until  these  conditions  are  corrected  will  prevent 

less  evaluation  and,  according  to  the  authors,  will  force  program 
improvements . 

Lewie,  Frank  L. ,  and  Frank  G.  Zarb.   Federal  program  evaluation  from  the  0MB 

perspective.  Public  administration  review,  v.  34,  July- Aug.  1974:  308-317. 

JK1.P85,  v.  34 
The  major  thrust  of  0MB' s  evalustion  role  in  the  immediate  future 
will  be  to  develop  the  collective  ability  to  generate  useful  evaluations 
which  are  relevant  to  policy  decisions,  taking  steps  to  assure  that  the 
evaluation  unit  is  adequately  staffed  and  has  a  close  relationship  with 
the  program  administrators. 

Marvin,  Keith  E.,  and  James  L.  Hedrick.   GAD  helps  Congress  evaluate  programs. 
Public  administration  review,  v.  34,  July- Aug.  1974:  327-333. 

ja.P85,  v.  34 
.r.ses  GAO's  role  in  program  evaluation,  in  program  results  audits, 
in  development  of  principles  and  standards  for  evaluation  and  in  develop- 
ment of  new  methods  of  evaluation  such  as  simulation  modeling. 

Poland,  Orville  F.   Program  evaluation  and  administrative  theory.   Public 
administration  review,  v.  34,  July- Aug.  1974:  333-338. 

JK1.P85,  v.  34 
Cites  the  "uneasy  position  of  program  evaluation  in  the  political 
realm"  as  program  evaluation  gives  high  priority  to  efficiency  and  effec- 
tiveness. 
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604 


Introduction 

In  these  days  when  large  organizations  and  huge  expenditures  of  money 
are  required  for  the  attempted  solution  of  many  social  problems,  it  is  to  be 
expected  that  public  policy  makers  would  rather  have  forewarning  of  anti- 
cipated problem  conditions  and  prepare  for  them  through  cogent,  well- 
designed  social  policies.   These  policies,  which  bear  on  the  comfort  and 
improved  well-being  of  the  people  insofar  as  they  are  affected  by  any  unit 
of  government,  can  range  from  a  plan  for  mass  rapid  transit,  to  school  mer- 
gers, to  encouraging  the  development  of  a  day  care  center. 

This  bibliography  cites  material  concerned  with  only  one  facet  of  the 
topic:  the  assessments  that  have  been  made  of  social  policies  through 
evaluation  research  techniques.   Evaluation  research  pursues  an  objective 
and  systematic  assessment  of  the  effectiveness  of  a  program  or  policy.   By 
using  comparative  studies,  computer  simulations,  projections,  quasi-control 
groups  and  other  techniques,  an  evaluator  can  choose  the  most  appropriate 
method  or  combination  of  methods  to  arrive  at  a  scientific  and  comprehen- 
sive assessment  of  a  given  policy. 

This  two-part  bibliography  presents  citations  to  journal  and  mono- 
graphic literature  on  the  broad  methodologies  of  evaluation  research  and 
its  theories  and  techniques  in  part  one;  and  in  part  two,  their  specific 
application  to  social  policy  formulations.  The  majority  of  the  citations  are 
annotated  and  have  been  chosen  from  the  computerized  bibliographic  data 
base  created  and  maintained  by  the  Library  Services  Division  of  the  Con- 
gressional Research  Service. 

The  material  cited  is  arranged  alphabetically  by  main  entry  within  each 
of  the  two  sections.   Citations  to  items  in  the  classified  collection  of  the 
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Library  of  Congress  are  provided  with  call  numbers,  items  designated  Law 
are  located  in  the  Law  Library  and  those  with  Newsp  are  found  in  the  News- 
paper and  Current  Periodical  Room  of  the  Library  of  Congress.  All  journal 
articles  are  provided  with  the  call  number  for  the  bound  volumes  of  the 
journal  regardless  of  whether  the  issues  cited  have  been  bound  at  this  time. 
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Abert,  James  B. ,  and  Murray  Kamrass,  eds.      Social  experiments 
and  social  program  evaluation;  proceedings  of  the  Washington 
Operations  Research  Council  symposium.     Cambridge,  Mass. , 
Ballinger  Pub.   Co.   [1974]    199  p.  HN57.A7 

Biderman,  Albert  D. ,   and  Laure  M.  Sharp.      The  competitive  evalua- 
tion research  industry.     Washington,  Bureau  of  Social  Science 
Research,  1972.     75  p. 

Examines  the  social  program  evaluation  research  "industry" 
which  has  recently  appeared  in  the  U.S.   and  the  competitive 
aspects  of  that  industry  (in  bidding  for  contracts,  etc. ). 

Buchanan,  Garth,  Pamela  Horst,   and  John  Scanlon.      Improving  Federal 
evaluation  planning.    Evaluation,  v.   1,  no.   2,  1973:  86-90. 

H62.A1E9,   v.   1. 
Describes  "an  approach  to  federal  evaluation  planning  that  the 
Urban  Institute  has  been  developing  in  work  for  the  Department 
of  Labor  and  the  National  Institute  of  Mental  Health.  " 

Davis,  Howard  R.      Four  ways  to  goal  attainment;  an  overview.     Evalu- 
ation,  v.   1,   no.    2.   1973:    43-48.  H62.  A1E9,   v.   1. 
Outlines  four  different  methods  of  goal  attainment  evaluation. 

Evaluation  research;  strategies  and  methods.  [Pittsburgh]  American 
Institutes  for  Research  [1970]    160  p.  H62.  5.U5E9 

Federal  evaluation  policy:    analyzing  the  effects  of  public  programs, 
by  Joseph  S.   Wholey  [and  others].     Washington,   Urban  Institute 
[1970]    134  p.  HN59.F37 

Partial  contents.  --Federal  evaluation- -what  it  is  and  why  it 
is  needed.  --Administration  of  an  evaluation  system.  --Organiza- 
tional relationships  and  responsibilities.  --Evaluation  resources. 
--Methodology. 

Bibliography:    p.   121-134. 

Gam,  Harvey  A.  ,   and  Mancur  Olson.       Public  services  on  the  assembly 
line.     Evaluation,   v.   1,   no.    2,   1973:    36-42,   96.  H62.A1E9,   v.   1. 

In  companion  articles,  the  authors  expand  on  evaluation  and  the 
care  that  must  be  exercised  when  measuring  public  performance. 

Gordon,  T.  J.,  S*  Enger,  and  R.  Rochberg.  An  experiment  in  simu- 
lation gaming  for  social  policy  studies.  Technological  forecasting, 
v.  L   Mar.   1970:    241-261.  T174.  T38,   v.  1 

"This  paper  describes  the  objectives,   design,   and  play  of  a 
simulation  game  developed  by  the  Institute  for  the  Future  under  a 
grant  from  the  Connecticut  Research  Commission.     The  general 
subject  of  the  research.  .  .was  the  design  of  methodologies  which 
promised  to  be  useful  in  forecasting  societal  change  and  the  applica- 
tion of  these  methodologies  to  derive  societal  forecasts  for  the 
state.  " 
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Guttentag,  Marcia.      Subjectivity  and  its  use  in  evaluation  research. 

Evaluation,  v.   1.  no.    2,   1973:     60-65.  H62.  A1E9,   v.  1. 

Describes  a  model  for  program  evaluation  which  quantifies 
decision-maker's  values  on  the  belief  that  the  inherent  subjectivity  of 
decision-making  makes  classical  experimental  designs  not  very  use- 
ful. 

Harberger,  Arnold  C.      Project  evaluation;  collected  papers.     Chicago, 
Markham  Pub.   Co.   [1972?]    330  p.  HD47.H28 

Hinrichs,  Harley  H.  ,   and  Graeme  M.   Taylor.     Systematic  analysis; 
a  primer  on  benefit-cost  analysis  and  program  evaluation.     Pacific 
Palisades,  Calif. ,  Goodyear  Pub.  Co.  [1972]    152  p. 

HJ2052.H53 

Lyden,  Fremont  James,  and  Lawrence  K.   Lee.      Evaluating  program 
change.     Social  work,   v.   18,   Mar.   1973:    87-94.  HV1.  S644,   v.  18. 

"The  authors  describe  a  methodology  developed  and  used  effectively 
in  one  metropolitan  area  to  measure  program  change.     They  identify 
and  utilize  the  findings  to  show  how  greater  change  might  be  effected 
in  more  agencies." 

Meinert,  Roland  G.      Futures  forecasting.     Social  work,   v.   18,   Nov.   1973: 
48-51.  HV1.  S644.   v.   18. 

"Four  systematic  techniques  of  forecasting  the  future,  widely 
practiced  in  other  fields,  could  be  useful  in  social  planning.  The 
techniques  are  extrapolation,  Delphi  forecasting,  simulation,  and 
scenario  speculation.  "  HV1.  S644,  v.   18. 

Symposium  on  policy  analysis.     Urban  affairs  quarterly,   v.   7,  June  1972: 
379-412.  HT101.U67,   v.   7 

Contents.  --Some  promising  approaches  to  policy  analysis:    an 
introduction,   by  J.   Brandl.  --An  advocate's  process  outline  for 
policy  analysis:    the  case  of  welfare  reform,  by  J.    Lyday.  --Micro- 
analytic  simulation  modeling  for  evaluation  of  public  formulation 
and  evaluation,   by  W.   Harrar  and  D.   Bawden.  --The  capacity  of 
social  science  organizations  to  perform  large-scale  evaluative 
research,   by  W.   Williams. 

Wegner,   Robert  E.   C.      Evaluating  the  evaluators.     Public  administra- 
tion review,   v.    33,   Jan. -Feb.   1973:    80-87.  JK1.  P85,  v.   33 

Weiss,   Carol  H.      Abstracts  on  evaluative  research,  by  Carol  H.  Weiss 
[and  others  [n.p.]  Model  Cities  Evaluation  Institute  [1973?]     1  v. 
(unpaged) 

Extensive  abstracts  of  articles  and  books  on  evaluation  in  a  variety 
of  program  areas  are  arranged  alphabetically  by  author  and  a  sub- 
ject/author index  is  provided. 
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—     Between  the  cup  and  the  lip. . .    Evaluation,  v.  1, 
no.    2.   1973:    49-55.  H62.A1E9,   v.  1 

"That  much  evaluation  research  falls  short  of  the  expectations 
of  funders,  program  staff,   and  evaluators  themselves  is  due  less 
to  limitations  of  research  expertise  than  to  organizational  contraints 
on  the  researcher's  ability  to  apply  what  he  knows.  " 


Evaluative  research:    methods  for  assessing  program  effective- 
ness.    Englewood  Cliffs,  N.  J. ,   Prentice -Hall  [1972]    160  p. 

H62.  W3962 

Wholey,  Joseph  S.      What  can  we  actually  get  from  program  evaluation? 
Policy  sciences,   v.   3,   Sept.   1972:    361-369.  HI.  P7,   v.   3 

"Assesses  the  role  program  evaluation  can  play  in  assisting 
decisions  on  public  programs.  " 

Wilemon,  David.      Transferring  space-age  management  technology. 
Conference  Board  record,  v.   7,  Oct.  1970:    50-55. 

HC101.C64,   v.    7 
"This  article  explores  the  characteristics  of  project  management 
to  appraise  its  transfer  potentialities  in  managing  society's  complex 
problems,  first  examining.  .  .  some  of  the  generalized  characteristics 
of  these  new  management  techniques.  " 
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Evaluations  of  Social  Policy 

Allen,  Gary.      America,  1973:    freedom  rings  for  help.    American 
opinion.  July -Aug.  1973:    1-15,  17-20.  AP2.04732 

Criticizes  the  political  establishment's  economic  and  social 
policies;  urges  a  freer  economy  and  stricter  law  enforcement. 

Bazell,   Robert  J.      NSF:    is  applied  research  at  the  take  off  point? 

Science,  v.  172,  June  25,  1971:    1315-1317.  A1S.  35,  v.  172 

Discusses  the  NSF's  new  RANN  (Research  Applied  to  National 
Needs)  program. 

Benefit-cost  analyses  of  Federal  programs;  a  compendium  of  papers. 
Submitted  to  the  Subcommittee  on  Priorities  and  Economy  in  Govern- 
ment of  the  Joint  Economic  Committee,  Congress  of  the  United 
States.    Washington,  U.  S.  Govt.   Print.  Off. ,  1972.     274  p. 

At  head  of  title:    92d  Congress,  2d  session.     Joint  Committee 
print.  HJ7539.B45 

Partial  contents.  --Report  on  a  survey  of  Federal  program 
evaluation  practices.  --Profiles  of  analytical  studies.  --The  family 
assistance  plan:    an  analysis  and  evaluation.  --Efficiency  and  equity 
effects  in  the  benefits  from  the  Federal  housing  program  in  1965. 
--Econometric  analysis  of  the  Medicare  program.  --Benefits  and 
costs  of  manpower  training  programs:    a  synthesis  of  previous  studies 
with  reservations  and  recommendations. 

Brezina,  Dennis  W.      The  congressional  debate  on  the  social  sciences 
in  1968.    Washington,   Program  of  Policy  Studies  in  Science  and 
Technology,  George  Washington  University,  1968.     21  p.     (George 
Washington  University,  Washington,  D.  C.    Staff  discussion  paper 
400) 

Campbell,  Donald  T.      Reforms  as  experiments.     Urban  affairs  quarterly, 
v.   7,  Dec.   1971:    133-171.  HT101.U67,   v.   7 

"The  United  States  and  other  modern  nations  should  be  ready  for 
an  experimental  approach  to  social  reform,  an  approach  in  which 
we  try  out  new  programs  designed  to  cure  specific  social  problems, 
in  which  we  retain.,  imitate,  modify,   or  discard  them  on  the  basis 
of  apparent  effectiveness  on  the  multiple  imperfect  criteria  avail- 
able."   See  also  "Problems  facing  Campbell's  'Experimental 
Society'"  by  Phillip  Shaver  and  Graham  Staines,  p.  173-192  in  the 
same  issue. 

Campfield,  William  L.      Financial  management  and  social  reporting 
by  the  Federal  Government  in  the  1970's.     Footnote,  no.   6,  winter 
1972-73:    1-14. 

Describes  some  of  the  grave  social  problems  that  Federal  man- 
agers and  their  accounting  and  other  professional  advisers  must 
confront,  focusing  on  the  constraints  and  limitations  on  social  pro- 
grams evaluation  as  well  as  selected  examples  of  efforts  by  Federal 
auditors  to  assess  the  effectiveness  of  social  program  planning  and 
performance. 
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Chapman,  William.      Scholars  examine  value  of  social  scientists  in  public 
policy.     Washington  post,   Feb.  18,  1973,  p.  A2.  Newsp 

Discusses  the  debate  about  the  role  of  social  scientists  in  public 
policy  involving  Adam  Yarmolinsky,  James  Q.  Wilson,  and  William  M. 
C apron. 

Clausen,  A.  W.      Toward  an  arithmetic  of  quality.     Conference  Board 
record,   v.   8,   May  1971:     9-13.  HC101.  C64,  v.   8 

The  first  step  toward  effective  policy  making  must  be  the  develop- 
ment of  a  set  of  social  accounts,  or  social  report,  that  can  "identify, 
assess  and  measure  those  elements  of  national  life  that  are  essential 
to  our  well-being—and  which  are  not,  and  cannot  be,  measured  by 
present  economic  indices." 

Coelho,  Goerge  V. ,  Elinor  Stillman,  and  Eli  A.  Rubinstein,  eds.      Social 
change  and  human  behavior;  mental  health  challenges  of  the  seventies. 
Rockville,  Md. ,  National  Institute  of  Mental  Health  [for  sale  by  the 
Supt.   of  Docs.,   U.S.   Govt.   Print.   Off.,   1972]    237  p.     (U.S. 
Dept.  of  Health,  Education,  and  Welfare.     DHEW  publication  no. 
(HSM)    72-1922) 

Partial  contents.  --Aggressive  behavior  and  human  evolution, 
by  S.  Washburn.  --Psychological  man  in  revolution:    the  struggle 
for  communal  resymbolization,  by  R.  Lifton.  --Roles  of  the  behavioral 
scientist  in  policy-oriented  research,  by  H.  Kelman.  --Science  policy 
and  the  government-university  partnership,  by  A.  Hunter  Dupree. 

Cohen,  Wilbur  J.      The  quality  of  Ufe  and  social  indicators.     National 
bureau  report,   Mar.  1972,   suppl.  :    1-11. 

Argues  for  greater  national  concern  with  social  problems  rather 
than  economic  gains. 

Cohn,  Elchanan.     Public  expenditure  analysis,  with  special  reference  to 
human  resources.     Lexington,   Mass.,   Lexington  Books  [1972]    157  p. 

HJ7451.  C56 

Cook,  Thomas  J. ,  and  Frank  P.  Scioli,  Jr.      A  research  strategy 
for  analyzing  the  impacts  of  public  policy.    Administrative  science 
quarterly,  v.  17,  Sept.   1972:    328-339.  HD28.A25,   v.  17 

Presents  "a  research  strategy  for  measuring  policy  impacts 
based  upon  the  principles  of  experimental  design  methodology.  " 

Duhl,   Leonard  J. ,  ed.      A  symposium  on  the  urban  crisis:    selected 

from  the  1968  Department  of  Housing  and  Urban  Development  summer 
study  in  urban  affairs.     Berkeley,  Center  for  Planning  and  Develop- 
ment Research,  University  of  California  [1970?]    458,   32  1. 

Partial  contents.  --New  roles  in  social  policy  planning.  --Social 
report  of  the  President.  --Social  indicators  of  minority  problems: 
the  case  of  Mexican-Americans. 
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Floyd,   Robert  H.      Federal  economic  policies  in  perspective.     Federal 
Reserve  Bank  of  Atlanta  monthly  review,  Apr.  1972:    62-67. 

Forrester,  Jay  W.      Counterintuitive  behavior  of  social  systems.     Tech- 
nology review,   v.   73,   Jan.   1971:    53-68.  T171.  M47,   v.   73 

'^his  paper  addresses  several  issues  of  broad  concern  in  the 
United  States:    population  trends;  the  quality  of  urban  life;  national 
policy  for  urban  growth;  and  the  unexpected,   ineffective,   or  detri- 
mental results  often  generated  by  government  programs  in  these 
areas. " 

Francis,  Walton.      What  social  indicators  don't  indicate.     Evaluation, 
v.  1,  no.   2,  1973:    79-83.  H62.  A1E9.   v.   1 

Freeman,  Howard  E. ,  and  Clarence  C.  Sherwood.  Social  research 
and  social  policy.  Englewood  Cliffs,  N.J.  ,  Prentice-Hall  [1970] 
159  p.     (Prentice-Hall  series  in  social  policy)  H62.  F72 

Glazer,  Nathan.      The  limits  of  social  policy.     Commentary,  v.   52, 

Sept.   1971:    51-58.  #  DS101.  C63,   v.    52 

Goodwin,   Leonard.      Bridging  the  gap  between  social  research  and 
public  policy:    welfare,   a  case  in  point.     Washington,   Brookings 
Institution,   1973.     114  p.      (Brookings  Institution,  Washington,   D.C. 
Reprint  266)  HV95.G59 

Suggests  that  an  approach  to  social  problems  called  experimental 
social  research  can  help  social  scientists  recognize  social  events 
as  social  problems. 

The  Great  society:    lessons  for  the  future:    Public  interest,  no.    34, 

winter  1974:    whole  issue.  HI.  P86,  no.   34 

Partial  contents.  --Social  intervention  in  a  democracy,  by  L. 
Liebman.  --Economic  policy  and  unemployment  in  the  1960's,  by  E. 
Phelps.  --Reform  follows  reality:    the  growth  of  welfare,   by  G. 
Steiner.  --What  does  it  do  for  the  poor?    A  new  test  for  national 
policy,  by  R.   Lampman.  --The  uses  and  limits  of  manpower  policy, 
by  L.   Ulman.  --Major  public  initiatives  in  health  care,  by  H. 
Klarman.  --Economic  developments  in  the  Black  community,  by 
A.   Brimmer.  --Blacks  and  the  crisis  of  political  participation,  by 
C.  Hamilton.  --Some  lessons  of  the  1960's,  by  E.  Ginzberg  and  R. 
Solow. 

Harman,  Willis  W. ,   and  John  T.   McAlister,   Jr.      An  assessment  of  the 
institutional  requirements  for  developing  and  implementing  national 
growth  policy.     In  Remarks  of  Vance  Hartke.     Congressional  record 
[daily  ed.]v.   119,  Nov.    27,   1973:    S21084-S21085.        J11.R5,   v.   119 

1  The  major  purpose  of  developing  explicit  national  growth  policy 
...  is  to  provide  a  framework  which  can  serve  as  a  basis  for  coordina- 
tion of  programs  intended  to  affect  growth,  and  that  can  ensure  that 
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growth  impacts  are  explicitly  considered  in  the  formulation  and  imple- 
mentation of  other  programs  whose  primary  purpose  is  not  directly 
related  to  growth. ' 

Holleb,  Doris  B.      Social  &  economic  information  for  urban  planning. 
Chicago,  Center  for  Urban  Studies,  University  of  Chicago,  1969. 
187  p.  HT167.H64 

Partial  contents.  --The  nature  of  an  information  system  for 
urban  planning.  --The  evolution  of  comprehensive  planning.  -- 
Metropolitan  perspectives  and  local  interests.  --Computers  and 
sophisticated  analytic  techniques. 

The  Human  being:    designer  or  designed?     Social  education,   v.    37, 

Nov.  1973:    622-634.  H62.A1S6,   v.   37 

Contents.  --Culture,   aggression,    and  where  we  are  going,  by 
P.  Bohannan.  --White  collar  conflict,  by  R.  Hammond. 

Koenig,   Louis  W.      The  President  needs  new  kinds  of  power.     In  Seven 
polarizing  issues  in  America  today.     Philadelphia,  American 
Academy  of  Political  and  Social  Science,  1971.     (Its  Annals,  v.   397, 
Sept.   1971)    p.   71-82  H1A4,  v.   397 

Dr.  Koenig  suggests  that  the  President  needs  to  increase  the 
variety  and  quality  of  his  power  to  create  a  better  social  policy. 

McVeigh,  Thomas.      Social  indicators:    a  bibliography.     Monticello, 

111.,   1971.     45  1.     (Council  of  Planning  Librarians.     Exchange  biblio- 
graph  215) 

Mishan,   Edward  J.      Economics  for  social  decisions;  elements  of  cost- 
benefit  analysis  [by]  E.   J.   Mishan.     New  York,   Praeger  [1973, 
C1972]    151  p.  HD47.  5.M532  1973 

Monti,  Lorna  A.      Social  indicators,  prices  and  politics:    uses  of 
planning  and  evaluation  data  in  Texas  state  government.     Texas 
business  review,  v.   48,   June  1974:    132-136.  HC107.T4T46,   v.   48 

Moynihan,  Daniel  P.       Policy  vs.   program  in  the  '70's.     Public  interest, 
no.    20,   summer  1970:    90-100.  HI.  P86,   no.   20 

Novick,   David.  -   Cost-benefit  analysis  and  social  responsibility.     Busi- 
ness horizons,   v.   16,  Oct.   1973:     63-72.  HF5001.  B828,   v.   16 

"Cost -benefit  analysis,  an  established  tool  for  choosing  the  path 
to  profit  maximization,  can  now  be  applied  to  decide  how  and  where 
to  use  funds  earmarked  for  a  'socially  responsible'  plan.  " 

Olson,   Mancur.      An  analytical  framework  for  social  reporting  and 
policy  analysis.     In  Political  intelligence  for  America's  future. 
Philadelphia,  American  Academy  of  Political  and  Social  Science, 
1970.     (Its  Annals,   v.    397,  Sept.   1971)    p.   112-126        H1.A4,   v.    397 
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Rossi,  Peter  Henry,  and  Walter  Williams,  eds.  Evaluating  social 
programs;  theory,  practice,  and  politics.  New  York,  Seminar 
Press,  1972.     326  p.     (Quantitative  studies  in  social  relations) 

H62.R664 
Partial  contents.  --Testing  for  success  and  failure  in  social 
action,  by  T.   R.  Houston.  --The  methodology  of  evaluating  social 
action  programs,  by  E.   L.   Dill  [and  others].  --Evaluating  Federal 
manpower  programs;  notes  and  observations,   by  D.   N.  Kershaw. 
--The  politics  of  evaluation:    the  case  of  Head  Start,  by  P.  H. 
Rossi. 

Salvatore,  Anthony  M.      Eight  steps  for  better  planning.     Human  needs, 
v.   1.  Apr. -May  1973:    29-31.  HV1.  H84,   v.  1 

"From  identifying  goals  to  monitoring  and  evaluating  results, 
each  step  in  the  social  service  planning  process  is  an  important 
one.  " 

Science,  public  policy  and  the  scientist  administrator;  an  anthology. 
Santa  Barbara,   Calif. ,  Center  for  the  Study  of  Democratic  Institu- 
tions [1971?]    265  p. 

The  articles  are  divided  into  three  sections.     "Part  I  deals  with 
the  socio-economic  implications  and  impact  of  science  and  tech- 
nology upon  society;  Part  II,  the  effect  of  national  advisory  and 
policy  groups  on  science  strategy  within  the  Executive  and  Legis- 
lative branches  of  government;  and  Part  III  presents  considerations 
of  the  fiscal  dilemma  of  academia  and  science  and  the  Federal  sup- 
port of  biomedical  research,  including  speculations  on  future  trends.  " 

Shapiro,   David  L.      Can  public  investment  have  a  positive  rate  of 
return?    Journal  of  political  economy,  v.   81,  Mar.  -Apr.  1973: 
401-413.  HB1.  J7,  v.   81 

"The  model  presented  assumes  that  public  managers  select 
investment  projects  that  maximize  the  net  private  benefits  of  their 
constituent  pressure  groups.     This  achieves  increased  agency 
size,  which  is  more  likely  to  increase  managerial  compensation 
than  to  maximize  net  social  benefits,   the  assumed  goal  of  the 
conventional  public  investment  model.   .  . .   The  analysis  as  well 
as  empirical  evidence  suggests  that  inefficient  public  projects  are 
the  rule  rather  than  the  exception.  " 

Sheldon,  Eleanor  Bernert,   and  Kenneth  C.   Land.      Social  reporting 
for  the  1970 's:    a  review  and  programming  statement.     Policy 
sciences,  v.   3,   July  1972:    137-151.  HI.  P7,   v.    3 

Tripodi,   Tony,   Phillip  Fellin,   and  Irwin  Epstein.      Social  program 
evaluation;  guidelines  for  health,   education,   and  welfare  adminis- 
trators.    Itasca,  111.,   F.   E.   Peacock  Publishers,   cl971.     142  p. 

HN29.T7 
Partial  contents.  --Demand  for  evaluation.  --Social  programs  and 
social  program  evaluation.  --Socio-political  context  of  evaluation. 
--Objectives  of  differential  evaluation.  --Techniques  of  evaluation. 
--Uses  of  consultation. 
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Evaluation  Research  and  Its  Application  to  Health  Programs: 
Selected  References,  1970-1974 

The  application  of  scientifically  oriented  social  research  techniques  to 
assess  a  program's  achievements  is  the  hallmark  of  evaluation  research. 
It  resembles  other  kinds  of  social  research  in  its  concern  for  accuracy, 
objectivity,  and  reliability  of  data,  but  it  differs  from  most  other  method- 
ologies in  that  it  is  usually  performed  in  a  dynamic  environment  over  which 
the  e valuator  has  little  or  no  control.  From  techniques  which  include 
comparative  studies,  computer  simulations,  projections  and  longitudinal 
evaluations,  the  researcher  must  choose  the  most  effective  method  or 
combination  of  methods  suitable  for  the  special  nature  of  the  program  under 
evaluation. 

This  bibliography  cites  published  literature  concerned  with  the  applica- 
tion of  evaluation  research  to  health  programs  in  the  United  States.  As 
the  national  health  authorities  deal  increasingly  with  interstate  health  prob- 
lems, programs  and  policies  have  been  developed  which  are  amenable  to 
evaluation  research.  In  similar  fashion,  the  operations  of  state  and  local 
health  programs  may  also    fall  within  the  purview  of  evaluation  research. 

The  following  two  part  bibliography  presents  citations  to  journal  and 
monographic  literature  on  the  broad  methodologies  of  evaluation  research 
and  its  theories  and  techniques  in  part  one;  and  in  part  two,  their  specific 
application  to  programs  in  the  national,  state,  and  local  health  areas.  This 
section  is  divided  into  four  topical  areas:  Public  Health  (which  includes 
health  facilities,  community  health  programs,  and  mental  health  care); 
Birth  Control  (which  includes  population  control,  child  health,  and  child 
care);  Alcoholism  and  Drug  Abuse  Treatment  Programs;  and  Medicare  and 
Medicaid. 

The  majority  of  the  citations  in  this  bibliography  have  been  selected 
from  the  computerized  bibliographic  data  base  created  and  maintained  by 
the  Library  Services  Division  of  the  Congressional  Research  Service.  The 
material  cited  is  arranged  alphabetically  by  main  entry  within  each  of  the 
sections.  Citations  to  items  in  the  classified  collections  of  the  Library 
of  Congress  are  provided  with  call  numbers  or  the  designation  Law  if  they 
are  in  the  Law  Library.  All  journal  articles  are  provided  with  the  call 
number  for  the  bound  volumes  of  the  journal  regardless  of  whether  the 
issues  cited  have  been  bound  at  this  time. 
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Abert.   James  B.  ,   and  Murray  Kamrass,  eds.      Social  experiments 
and  social  program  evaluation;  proceedings  of  the  Washington 
Operations  Research  Council  symposium.     Cambridge,  Mass., 
Ballinger  Pub.  Co.   [1974]    199  p.  HN57.A7 

Biderman,  Albert  D. ,  and  Laure  M.  Sharp.      The  competitive  evalua- 
tion research  industry.     Washington,  Bureau  of  Social  Science 
Research,  1972.     75  p. 

Examines  the  social  program  evaluation  research  "industry" 
which  has  recently  appeared  in  the  U.S.  and  the  competitive 
aspects  of  that  industry  (in  bidding  for  contracts,  etc.  ). 

Buchanan,  Garth,  Pamela  Horst,  and  John  Scanlon.      Improving  Federal 
evaluation  planning.     Evaluation,  v.   1,   no.    2,   1973:  86-90. 

H62.A1E9,   v.   1. 
Describes  "an  approach  to  federal  evaluation  planning  that  the 
Urban  Institute  has  been  developing  in  work  for  the  Department 
of  Labor  and  the  National  Institute  of  Mental  Health. " 

Carmichael,  Nancy,  and  Robert  Parke.      Information  services  for 
social  indicators  research.    Special  libraries,  v.   65,  May- June 
1974:    209-215.  Z671.S72,   v.   65 

Describes  the  needs  of  researchers  concerned  with  the  measure 
of  social  change,   "mentions  some  of  the  sources  which  exist  to  meet 
them,  and  shows  how  the  Center  for  Coordination  of  Research  on 
Social  Indicators  functions  as  an  information  clearinghouse  and  com- 
munication center  to  help  researchers  find  the  information  they 
require. " 

Davis,  Howard  R.      Four  ways  to  goal  attainment;  an  overview.    Evalu- 
ation, v.   1,   no.   2.   1973:    43-48.  H62.A1E9,   v.   1. 
Outlines  four  different  methods  of  goal  attainment  evaluation. 

Evaluation  of  social  intervention.    San  Francisco,  Jossey-Bass,  1972. 
267  p.     (The  Jossey-Bass  behavioral  science  series) 

HV40.E9 
Proceedings  of  a  symposium  held  Jan.  14-15,  1971  at  the  Lincoln 
Center  Campus  of  Fordham  University. 
Bibliography:    p.   255-262. 

Evaluation  research;  strategies  and  methods.     [Pittsburgh]    American 
Institutes  for  Research  [1970]    160  p.  H62.  5.  U5E9 

Federal  evaluation  policy:    analyzing  the  effects  of  public  programs, 
by  Joseph  S.  Wholey  [and  others].    Washington,   Urban  Institute 
[1970]    134  p.  HN59.F37 

Partial  contents.  --Federal  evaluation- -what  it  is  and  why  it 
is  needed.  — Administration  of  an  evaluation  system. — Organiza- 
tional relationships  and  responsibilities. --Evaluation  resources. 
-  -Methodology . 

Bibliography:    p.   121-134. 
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Gam,   Harvey  A. ,   and  Mancur  Olson.       Public  services  on  the  assembly 
line.     Evaluation,  v.   1,   no.   2,   1973:    36-42,   96.  H62.A1E9,   v.   1. 

In  companion  articles,  the  authors  expand  on  evaluation  and  the 
care  that  must  be  exercised  when  measuring  public  performance. 

Gordon,  T.  J. ,  S.  Enger,  and  R.  Rochberg.  An  experiment  in  simu- 
lation gaming  for  social  policy  studies.  Technological  forecasting, 
v.   1,   Mar.   1970:    241-261.  T174.T38,  v.   1 

"This  paper  describes  the  objectives,   design,  and  play  of  a 
simulation  game  developed  by  the  Institute  for  the  Future  under  a 
grant  from  the  Connecticut  Research  Commission.     The  general 
subject  of  the  research.  .  .was  the  design  of  methodologies  which 
promised  to  be  useful  in  forecasting  societal  change  and  the  applica- 
tion of  these  methodologies  to  derive  societal  forecasts  for  the 
state." 

Guttentag,   Marcia.      Subjectivity  and  its  use  in  evaluation  research. 
Evaluation,  v.   1,  no.    2,   1973:     60-65.  H62.A1E9,   v.   1 

Describes  a  model  for  program  evaluation  which  quantifies 
decision-maker's  values  on  the  belief  that  the  inherent  subjectivity  of 
decision-making  makes  classical  experimental  designs  not  very  use- 
ful. 

Harberger,  Arnold  C.  Project  evaluation;  collected  papers.  Chicago, 
Markham  Pub.   Co.   [1972?]    330  p.  HD47.H28 

Hinrichs,  Hariey  H. ,  and  Graeme  M.  Taylor.     Systematic  analysis; 
a  primer  on  benefit-cost  analysis  and  program  evaluation.     Pacific 
Palisades,  Calif.,  Goodyear  Pub.  Co.   [1972]    152  p. 

HJ2052.H53 
Simplifies  and  summarizes  the  core  of  systems  analysis  pre- 
senting case  studies  and  successful  examples  of  completed  analysis, 
illustrating  different  analysis  techniques. 

Lyden,   Fremont  James,   and  Lawrence  K.   Lee.      Evaluating  program 
change.     Social  work,   v.   18,   Mar.  1973:    87-94.  HV1.S644,   v.   18 

"The  authors  describe  a  methodology  developed  and  used  effectively 
in  one  metropolitan  area  to  measure  program  change.     They  identify 
and  utilize  the  findings  to  show  how  greater  change  might  be  effected 
in  more  agencies.  " 

Meinert,  Roland  G.      Futures  forecasting.    Social  work,  v.  18,   Nov.  1973: 
48-51.  HV1.S644,   v.   18 

"Four  systematic  techniques  of  forecasting  the  future,  widely 
practiced  in  other  fields,  could  be  useful  in  social  planning.  The 
techniques  are  extrapolation,  Delphi  forecasting,  simulation,  and 
scenario  speculation." 
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Moursund,   Janet.      Evaluation:    an  introduction  to  research  design. 
Monterey,  Calif.,  Brooks/ Cole  Pub.  Co.   [1973]    153  p. 

H62.M68 
Bibliography:    p.   148-150. 

Symposium  on  policy  analysis.     Urban  affairs  quarterly,  v.   7,  June  1972: 
379-412.  HT101.U67,   v.   7 

Contents.  — Some  promising  approaches  to  policy  analysis:    an 
introduction,  by  J.  Brandl.  --An  advocate's  process  outline  for 
policy  analysis:    the  case  of  welfare  reform,   by  J.   Lyday.  --Micro - 
analytic  simulation  modeling  for  evaluation  of  public  formulation 
and  evaluation,  by  W.  Harrar  and  D.  Bawden.  --The  capacity  of 
social  science  organizations  to  perform  large-scale  evaluative 
research,  by  W.  Williams. 

Wegner,  Robert  E.  C.      Evaluating  the  evaluators.     Public  administra- 
tion review,   v.   33,  Jan. -Feb.  1973:    80-87.  JK1.  P85,   v.   33 

Weiss,  Carol  H.      Abstracts  on  evaluative  research,  by  Carol  H.  Weiss 
[and  others,     n.  p.  ]  Model  Cities  Evaluation  Institute  [1973?]  1  v. 
(unpaged) 

Extensive  abstracts  of  articles  and  books  on  evaluation  in  a  variety 
of  program  areas  are  arranged  alphabetically  by  author  and  a  sub- 
ject/author index  is  provided. 


Between  the  cup  and  the  lip. . .    Evaluation,  v.  1, 
no.    2,   1973:    49-55.  H62.A1E9,   v.  1 

"That  much  evaluation  research  falls  short  of  the  expectations 
of  funders,  program  staff,   and  evaluators  themselves  is  due  less 
to  limitations  of  research  expertise  than  to  organizational  contraints 
on  the  researcher's  ability  to  apply  what  he  knows. " 


Evaluative  research:    methods  for  assessing  program  effective- 
ness.    Englewood  Cliffs,   N.  J.,   Prentice-Hall  [1972]    160  p. 

H62.W3962 

Wholey,  Joseph  S.      What  can  we  actually  get  from  program  evaluation? 
Policy  sciences,  v.    3,  Sept.   1972:    361-369.  HI.  P7,   v.    3 

"Assesses  the  role  program  evaluation  can  play  in  assisting 
decisions  on  public  programs.  " 

Wilemon,  David.      Transferring  space-age  management  technology. 
Conference  Board  record,  v.   7,  Oct.  1970:    50-55. 

HC101.C64,  v.  7 
"This  article  explores  the  characteristics  of  project  management 
to  appraise  its  transfer  potentialities  in  managing  society's  complex 
problems,  first  examining. . .  some  of  the  generalized  characteristics 
of  these  new  management  techniques.  " 
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Berkowitz,  Monroe.  New  emphasis  on  evaluation  for  services  management.  Social 
and  rehabilitation  record,  v.  1,  July-Aug.  1974:  20-23. 

"It  is  truly  possible  to  evaluate  but  not  to  denigrate,  to  be  rational 
without  being  callous,  to  be  accountable  to  the  taxpayer  without  sacrific- 
ing responsibility  to  the  handicapped.  But  this  can  be  done  only  if  goals 
are  shaped  which  challenge  those  who  build  evaluation  models  to  join  in 
reaching  towards  them." 

Downing,  Billie,  and  others.  Accountability  in  administration:  responsibilities 
to  clients,  staff  and  the  public.  Mental  retardation,  v.  12,  June  1974: 
10-12.  RC570.M38,  v.  12 

"New  emphasis  is  being  placed  on  accountability  in  programming,  budget- 
ing and  planning  throughout  the  continuum  of  services  for  the  mentally 
retarded.  Administrators  are  being  called  upon  to  recognize  increasing 
demands  on  the  tax  dollar,  the  need  for  objective  control  of  program  quali- 
ty, and  priority  relationship  of  mental  retardation  services  to  other 
social-health  problems.  The  article  discusses  administrative  responsibility 
to  clients,  staff  and  the  public." 

Epstein,  Leon  M. ,  and  others.  Evaluation  of  a  program  of  aftercare  for  patients 
discharged  from  the  hospital.  Medical  care,  v.  11,  July-Aug.  1973:  320-327. 

RA413.M397,  v.  11 

^Evaluation  of  Federal  health  programs.  GAO  review,  summer  1973:  72-74. 

HJ9701.63 

Feldman,  Saul,  and  Charles  Windle.  The  NIMH  approach  to  evaluating  the  community 
mental  health  centers  program.  Health  services  reports,  v.  88,  Feb.  1973: 
174-180.  RA445.A25,  v.  0C 

"The  NIMH  evaluation  approach  consists  of  the  coordination  of  four  major 
components—monitoring,  enhancing  the  evaluation  capability  of  community 
mental  health  centers,  program  description,  and  evaluation  of  the  national 
program." 

Goltz,  Barbara,  Thomas  N.  Rusk,  and  Richard  A.  Sternbach.  A  built-in  evaluation 
system  in  a  new  community  mental  health  program.  American  journal  of  public 
health,  v.  63,  Aug.  1973:  702-709.  RA421.A41,  v.  63 

"The  evaluation  of  a  new  community  mental  health  service  is  discussed 
in  terms  of  the  design  of  the  evaluation  system,  the  goals  of  the  program, 
the  therapy  system,  and  lastly  staff  reactions  and  problems  concerned  with 
the  procedures." 
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Grimshaw,  Thomas  E.,  Kenneth  F.  Nolan,  Jr.,  and  Steven  C.  Brady.  Why  program  cost- 
ing works  for  a  medical  center,  and  why  it  might  not  work  for  you.  College 
&  university  business,  v.  55,  Nov.  1973:  36-39.         LB2300.C5,  v.  55 

Grosse,  Robert  N.  Cost-benefit  analysis  of  health  service.  In  The  Nation's  health: 
some  issues.  Philadelphia.  American  Academy  of  Political  and  Social  Science, 
1972.   (Its  Annals,  v.  399,  Jan.  1972)  p.  89-99.         H1.A4,  v.  399 

The  author  believes  that  "the  costs  of  health  services  should  be  a  sig- 
nificant consideration  in  deciding  upon  governmental  health  policies  and 
programs."  He  gives  examples  of  cost  effectiveness  based  on  dollars  ex- 


pended and  lives  saved. 

Hardwick,  C.  Patrick,  and  Harvey  Wolfe.  An  incentive  reimbursement/industrial 
engineering  experiment.  Rockville,  Md.,  U.S.  Health  Services  and  Mental 
Health  Administration  [available  from  NTIS]  1970.  537  p. 

"PB-204  924" 

"The  evaluation  focuses  on  quantification  of  cost  savings,  obstacles 
to  full  implementation,  the  incentive  calculations  and  their  effect,  and 
how  involvement  in  the  program  affected  Blue  Cross  and  participating 
hospitals." 

Harwell,  Mallory.  A  study  on  the  accessibility  of  community  mental  health  centers; 
final  report.  [Cambridge,  Mass.,  Abt  Associates]  1972.  182  p. 

"PB  213  117" 

"Two  major  types  of  accessibility  'barriers'  were  considered:  psycho- 
logical barriers  (people's  awareness  of  and  attitudes  toward  services)  and 
physical  barriers  (the  location  cost,  and  other  operational  characteristics 
of  services)." 

Health  facilities  and  health  services.  Inquiry,  v.  10,  Mar.  1973:  17-39. 

RA410.A1I58,  v.  10 
Ray  E.  Brown  presents  a  paper  examining  the  role  of  state  government  in 
health  facilities  and  services  where  the  state  government  has  superior  capa- 
bilities. Melvin  A.  Glasser,  J.  0.  Wallace,  Phillip  G.  Thompson  and  William 
Stonemen  write  the  comments,  criticisms  and  analysis  that  follow  the  main 
article. 

Hodges,  Allen.  Matching  resources  with  your  priorities.  Social  and  rehabilitation 
record,  v.  1,  Feb.  1974:  6-9. 

Presents  a  strong  case  for  funding  public  health  projects  according  to 
their  priority  and  for  follow-up  evaluation  projects. 

Holland,  Winford  E.  and  Harrison  H.  Huntoon.  The  evaluation  of  experimental  social 
service  delivery  systems  at  the  community  level:  an  organizational  effective- 
ness view.  Community  mental  health  journal,  v.  10,  spring  1974:  41-51. 

RA790.A1C53,  v.  10 
Discusses  in  very  general  terms,  issues  relevant  to  the  evaluation  of 
experimental  or  pilot  programs  at  the  community  level.  "Five  key  issues  are 
identified  and  discussed:  (1)  a  concept  of  organizational  effectiveness;  (2) 
the  role  of  the  evaluation  function;  (3)  the  political  environment  for  evalu- 
ation; (4)  organizational  configurations  for  evaluation;  and  (5)  generalized 
steps  for  the  implementation  of  the  evaluation  subsystem." 
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Hornstra,  Robijn  K.,  and  Bess  Udell.  A  measure  of  delivery  patterns  in  psychiatric 
care.  Community  mental  health  journal,  v.  10,  spring  1974:  84-88. 

RA790.A1C53,  v.  10 
"The  ratio  of  a  1-year  applicant  count  over  a  1-day  caseload  count  is 
presented  as  a  measure  of  delivery  patterns  or  conversely,  utilization  of 
services.  These  counts  can  be  easily  obtained.  The  practical  use  of  the 
measure  is  illustrated  by  comparing  the  ratios  of  various  facilities,  of 
major  diagnostic  groups,  and  as  an  index  of  change  of  delivery  patterns." 

Kakalik,  James  S.,  and  others.  Services  for  handicapped  youth:  a  program  over- 
view. Santa  Monica,  Calif.,  Rand,  1973.  341  p. 

Describes  "a  comprehensive,  22-month  cross  agency  evaluation  of  Federal 
and  state  programs  for  assistance  to  handicapped  youth." 

Kanno,  Charles  K.  Eleven  indices:  an  aid  in  reviewing  state  and  local  mental 
health  and  hospital  programs.  Washington,  Joint  Information  Service  of  the 
American  Psychiatric  Association  and  the  National  Association  for  Mental 
Health,  1971.  83  p. 

Kleffel,  Dorothy.  A  statistical  information  program  for  utilization  review  in 
extended  care  facilities.  [Washington]  U.S.  Health  Services  and  Mental 
Health  Administration,  Community  Health  Service  [for  sale  by  the  Supt.  of 
Docs.,  U.S.  Govt.  Print.  Off.]  1972.  65  p.  (U.S.  Dept.  of  Health,  Educa- 
tion and  Welfare.  DHEW  publication  no.  HSM  72-6500) 


A  utilization  review  program  for  home  health  agencies.  [Washington]  U.S. 

Health  Services  and  Mental  Health  Administration,  Community  Health  Service 
[for  sale  by  the  Supt.  of  Docs.,  U.S.  Govt.  Print.  Off.]  1972.  35  p. 
(U.S.  Dept.  ot  Health,  Education,  and  Welfare.  DHEW  publication  no.  HSM 
72-6502) 

Presents  guidelines  and  suggestions  for  developing  a  technique  to 
evaluate  patient  care. 

Klerman,  Gerald.  Current  evaluation  research  on  mental  health  services.  Ameri- 
can journal  of  psychiatry,  v.  131,  July  1974:  783-787. 

RC321.A52,  v.  131 
Surveys  "five  major  sources  of  systematic  evaluations  of  mental  health 
services:  1)  the  public  at  large,  2)  governmental  agencies,  3)  the  courts, 
4)  the  psychiatric  profession,  and  5)  the  Nixon  Administration.  The  changing 
relationship  between  the  mental  health  professions  and  the  society  at  large 
will  be  the  major  focus  of  attention." 

Levin,  A.L.  Health  planning  and  the  U.S.  federal  Government.  International  jour- 
nal of  health  services,  v.  2,  Aug.  1972:  367-376.        RA421.I49,  v.  2 

"The  degree  to  which  planning  is  applicable  to  current  health  problems, 
including  those  of  rising  prices  and  misuse  of  resources,  and  future  federal 
reactions  to  these  problems,  are  commented  upon." 
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Lombillo,  Jose  R.,  Thomas  J.  Kirasuk,  and  Robert  E.  Sherman.  Evaluating  a  community 
mental  health  program  contract  fulfillment  analysis.  Hospitals  community 
psychiatry,  v.  24,  Nov.  1973:  760-762.  RC443.A1M4,  v.  24 

"A  small  community  mental  health  program  evaluates  its  effectiveness 
through  a  treatment  outcome  measurement  built  into  the  day-to-day  delivery  of 
outpatient  services." 

Mannino,  Fortune  V.,  and  Milton  F.  Shore.  Demonstrating  effectiveness  in  an  after- 
care program.  Social  work,  v.  19,  May  1974:  351-354. 

HV1.S644,  v.  19 
Suggests  that  social  workers  need  to  improve  their  evaluation 
techniques. 

Martz,  Helen,  and  John  Trauth.  Six  steps  to  measure  change.  Human  needs,  v.  1, 
Feb.  1973:  25-27.  HV1.H84,  v.  1 

Describes  the  Helena  Comprehealth  Plan  which  is  designed  to  test  the 
feasibility  of  a  prepaid  health  care  program  for  the  'medically  needy.' 

Moore,  John  H.  The  development  of  mental  health  resources  in  four  pairs  of  coun- 
ties between  1958  and  1970.  New  York,  National  Study  Service  [distributed  by 
NTIS]  1972.  268  p. 

"PB  210  027" 

Traces  "the  significant  development  factors  leading  to  changes  in  mental 
health  resources  in  two  pairs  of  counties:  namely,  Rutland  County,  Vermont, 
and  Whatcom  County,  Washington;'and  Schenectady  County,  New  York,  and  Ingham 
County,  Michigan,  between  1958  and  1970  (only  one  of  each  pair  received  a 
Federal  staffing  grant  for  a  CMHC).  There  was  noted  substantial  progress  in 
the  development  of  mental  health  resources  in  all  four  counties,  but  progress 
in  the  two  counties  with  Federal  funding  has  been  more  substantial." 

Nelson,  Bardin  H. ,  Jr.,  and  J.  Robert  Pecarchik.  A  computerized  program  audit  for 
community  mental  health  clinics.  Community  mental  health  journal,  v.  10, 
spring  1974:  102-110.  RA790.A1C53,  v.  10 

Describes  a  prototype  computerized  data  processing  system  for  conducting 
accurate  and  meaningful  evaluations  of  the  operation  and  services  delivery  of 
community  mental  health  clinics. 

Rappaport,  Maurice.  Evaluating  community  mental  health  services:  guidelines  for 
an  administrator.  Hospital  &  community  psychiatry,  v.  24,  Nov.  1973: 
757-760.  RC443.A1M4,  v.  24 

Addresses  factors  in  assessing  costs  and  effectiveness,  unusual 
research  strategies,  and  the  need  for  adequate  support  for  evaluation 
research. 
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Stoddart,  Greg  L.  Effort-reporting  and  cost  analysis  of  medical  education.  Jour- 
nal of  medical  education,  v.  48,  Sept.  1973:  814-823. 

R11.A94,  v.  48 
Presents  an  historical  review  of  applications  of  effort-reporting  and 
describes  a  two-step  process  which  will  address  the  estimating  of  joint- 
production  problems  and  introduces  enough  flexibility  to  accommodate  a 
multidisciplinary  medical  education  setting. 

Sullivan,  Daniel  F.  Conceptual  problems  1n  developing  an  index  of  health.  Rock- 
ville,  Md.,  U.S.  National  Center  for  Health  Statistics  [for  sale  by  the  Supt. 
of  Docs.,  U.S.  Govt.  Print.  Off.,  1974]  18  p.  RA409.U45  no. 17 

(U.S.  Dept.  of  Health,  Education,  and  Welfare.  DHEW  publication  no.  (HRA) 
74-1017)  (U.S.  National  Center  for  Health  Statistics.  Vital  and  health 
statistics.  Series  2:  Data  evaluation  and  methods  research,  no.  17) 

Considers  the  essential  characteristics  of  an  index  using  measures 
indicative  of  changing  health  status  in  the  United  States  population,  with 
a  review  of  previous  studies  of  general  health  indices.  Problems  in  de- 
ciding what  to  measure  are  discussed  and  a  method  of  measurement  is  described. 

Thorn,  Anne-Marie,  and  Katherine  Stafford.  Time  for  evaluation  of  services.  Hos- 
pitals, v.  46,  Oct.  1,  1972:  57-60.  RA960.H6,  v.  46 

Outlines  possible  evaluation  techniques  that  can  be  used  to  measure  the 
effectiveness  ot  home  medical  care. 

U.S.  Dept.  ot  Health,  Education,  and  Welfare.  A  study .of  national  health  insur- 
ance proposals  introduced  in  the  92nd  Congress;  a  supplementary  report  to 
the  Congress.  [Washington]  1971.  83  1. 

"The  purpose  of  the  study  reported  here  is  essentially  twofold:  1.  To 
develop  and  to  present  a  methodology  by  which  the  costs  of  various  national 
health  insurance  proposals  can  be  estimated.  2.  To  apply  the  methodology 
to  the  various  bills  introduced  in  the  92nd  Congress  prior  to  April  30,  1971.' 

Zemach,  Rita.  Program  evaluation  and  system  control.  American  journal  of  public 
health,  v.  63,  July  1973:  607-609.  RA421.A41,  v.  63 

"Evaluation  of  health  programs  1s  considered  in  terms  of  system  control 
theory,  as  a  process  within  a  system  linked  to  decision-making  levels,  and 
using  both  feedback  and  forward-feeding  of  information.  This  requires  plan- 
ning the  evaluation  process  in  planning  the  program,  in  terms  of  prevailing 
requirements  and  constraints." 
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Evaluations  of  Birth  Control,  Population  and  Child  Care  Programs 


Boaz,  Rachel  Floersheim.  Free-standing  subsidized  family  planning  clinics:  the 
manpower  cost  of  patient  services.  Inquiry,  v.  10,  Mar.  1973:  14-25. 

RA410.A1I58,  v.  10 

Bogue,  Donald.  Cost-effectiveness  analysis  of  family  planning  programs.  [Chicago] 

Community  and  Family  Study  Center,  University  of  Chicago,  1973.  Ill  p. 

(Rapid  Feedback  for  Family  Planning  Improvement.  Family  planning  research 
and  evaluation  manual  no.  11) 


Family  planning  improvement  through  evaluation;  a  manual  of  basic  principles, 

Donald  J.  Bogue,  editor.  [Chicago]  Community  and  Family  Study  Center,  Univer- 
sity of  Chicago,  1970.  82  p.  (Family  planning  research  and  evaluation 
manual,  no.  1)  HQ763.5.B63 

Burt,  Marvin  R.,  and  Louis  L.  Blair.  Options  for  improving  the  care  of  neglected 
and  dependent  children;  program  analysis  applied  to  local  government. 
Washington,  Urban  Institute,  1972.  134  p.  HV743.N2B87 

Chokel ,  Patricia  K. ,  and  Janet  T.  Dingle.  Using  standard  data  for  program  evalu- 
ation in  Cleveland.  Family  planning  perspectives,  v.  5,  fall  1973:  216-222. 

HQ763.  F33,  v.  5 
Outlines  a  method  of  analysis  for  administrators  who  wish  to  measure  the 
Impact  of  their  population  control  programs.  The  types  of  data  used  in  this 
analysis  are  simple,  straightforward,  and  readily  available. 

Costello,  Joan,  and  Eleanor  Binstock.  Review  and  summary  of  a  national  survey  of 
the  parent-child  center  program.  [Washington,  U.S.  Bureau  of  Head  Start  and 
Early  Childhood,  1971]  47  p.  HV741.C66 

Prepared  for  the  Office  of  Child  Development  from  the  report  submitted 
by  Kirschner  Associates. 

Forrester,  Jay  W.  Counterintuitive  behavior  ot  social  systems.  Technology  review, 
v.  73,  Jan.  1971:  53-68.  T171.M47,  v.  73 

"This  paper  addresses  several  issues  of  broad  concern  1n  the  United 
States:  population  trends;  the  quality  of  urban  life;  national  policy  for 
urban  growth;  and  the  unexpected,  ineffective,  or  detrimental  results  often 
generated  by  government  programs  in  these  areas." 

Goldstein,  Hyman.  Observations  on  evaluation  of  family  planning  programs.  Health 
services  reports,  v.  88,  Mar.  1973:  213-217.  RA445.A25,  v.  88 

Hamilton,  Marshall  L.  Evaluation  of  a  parent  and  child  center  program.  Child 
welfare,  v.  51,  Apr.  1972:  248-258.  .    HV701.C52,  v.  51 
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Herman,  Hannah,  and  Dorothy  C.  Adkins.  Hawaii  Head  Start  evaluation  —  1968-69; 
final  report.  Honolulu,  Education  Research  and  Development  Center,  Univer- 
sity of  Hawaii,  1970.  182  1.  LC4092.H3H4 

At  head  of  title:  University  of  Hawaii.  Head  Start  Evaluation  and 
Research  Center. 

Hu,  Teh-wei .  Selected  bibliography  on  child  care  evaluation  studies.  Monticello, 
111.,  1973.  9  p.  (Council  of  Planning  Librarians.  Exchange  bibliography 
440) 

A  cost  analysis  of  day  care  centers  in  Pennsylvania  [by]  Teh-wei  Hu  [and] 

Karl  Wise.  University  Park,  College  of  Human  Development,  Pennsylvania 
State  University,  1973.  47  p.  HV857.P4H8 

Kelly,  William  John.  A  cost-effectiveness  study  of  clinical  methods  of  birth 
control:  with  special  reference  to  Puerto  Rico.  New  York,  Praeger 
Publishers  [1972,  cl971]  112  p.  (Praeger  special  studies  in  international 
economics  and  development)  HQ766.5.P8K44 

McLaughlin,  Curtis  P.,  and  Edward  S.  Trainer.  Qualitative  evaluation  of  family 
planning  proposals  and  programs:  a  systems  approach.  Popsim,  a  demographic 
microsimulation  model  [by]  D.  G.  Horvitz  [and  others.  Chapel  Hill,  N.C.] 
Carolina  Population  Center,  1971.  70  p.  (Carolina  Population  Center. 
Monograph  no.  12)  HQ766.M22 

Includes  bibliographies. 

Nesenholtz,  David,  and  Jurgen  Schmandt.  Social  policy  and  the  new  federalism: 
the  case  of  child-development  policy.  Public  affairs,  v.  19,  Aug.  1973: 
1-6.  JK6501.P83,  v.  19 

Discusses  the  Federal  influence  on  Texas  child  development  policies  and 
programs. 

Roby,  Pamela  A.  Child  care-who  cares?  Foreign  and  domestic  infant  and  early 
childhood  development  policies.  New  York,  Basic  Books  [1973]  456  p. 

HV854.R6 

Rosengard,  Barbara,  ed.  Research,  demonstration,  and  evaluation  studies,  fiscal 
year  1973.  [Washington]  U.S.  Children's  Bureau,  Office  of  Child  Develop- 
ment [1973]  71  p.  (U.S.  Dept.  of  Health,  Education,  and  Welfare.  DHEW 
publication  no.  (OCD)  74-30) 

Partial  contents. --Day  care. --Early  childhood. --Adoption  and  foster 
care. --Children's  institutions. --Education  for  parenthood. --Single-parent 
families. --Head  Start  evaluation  projects. 
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Rosoff,  Jeannie  I.  Can  state-administered  programs  achieve  the  national  goal? 
Family  planning  perspectives,  v.  5,  fall  1973:  209-212. 

HQ763.F33,  v.  5 
Concludes  that  without  a  considerable  shift  in  state  priorities, 
Federal  funds  will  probably  not  be  sought  and  used  for  family  planning 
programs. 

Shultz,  Carl  S.,  and  Marilyn  L.  Martin.  Direct  and  indirect  financing  of  family 
planning  programs:  project  grants  and  reimbursement  mechanisms.  Family 
planning  perspectives,  v.  5,  fall  1973:  202-208.         HQ763.F33,  v  5 

Presents  "an  analysis  of  the  likely  consequences  of  one  proposed  mech- 
anism for  implementing  this  policy:  namely,  tying  the  financing  of  a 
program  of  health  services  delivery—family  planning—into  state-adminis- 
tered public  welfare  systems,  in  this  case  through  SSA  Titles  XIX  and  IV-A." 

Simon,  Julian  L.  Toward  formal  valuation  of  scientific  research  projects:  a 

case  study  in  population  control.  Policy  sciences,  v.  3,  July  1972:  177-181. 

H1.P7,  v.  3 

Sparer,  Gerald,  Louise  M.  Okada,  and  Stanley  Till inghast.  How  much  do  family 
planning  programs  cost?  Family  planning  perspectives,  v.  5,  spring  1973: 
100-106.  HQ763.F33,  v.  5 

Westinghouse  Learning  Corporation.  Day  care  survey,  1970;  summary  report  and 
basic  analysis,  by  Westinghouse  Learning  Corporation  and  Westat  Research 
Inc.   Prepared  for:  Evaluation  Division,  Office  of  Economic  Opportunity. 
[Bladensburg,  Md.l  1971—      v.  HV854.W45 

Young,  David  W.  Case  costing  in  child  care:  a  critical  step  toward  increased 
accountability  in  social  services.  Child  welfare,  v.  52,  May  1973: 
299-304.  HV701.C52,  v.  52 

Zaidan,  George  C.  The  costs  and  benefits  of  family  planning  programs.  Baltimore, 
Johns  Hopkins  Press  [1971]  52  p.  (World  Bank  staff  occasional  papers,  no.  12] 

HQ766.Z3 


Evaluations  of  Alcoholism  and  Drug  Abuse  Treatment  Programs 


Alaska.  Office  of  Alcoholism.  Alcoholism  in  Alaska  and  the  alcoholism  grant-in- 
aid  program.  Juneau,  1971.  83  p.  HV5297.A4A45 

Alternative  approaches  to  opiate  addiction  control:  costs,  benefits  and  poten- 
tial tby]  William  H.  McGlothin  [and  others]  Washington,  Drug  Control  Division, 
Bureau  of  Narcotics  and  Dangerous  Drugs,  1972.  65,  A22,  B35  p. 

HV5816.A45 
"SCID  TR-7" 
Includes  bibliographies. 
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Berecochea,  John  E.,  and  George  E.  Sing.  The  effectiveness  of  a  halfway  house  for 
civilly  committed  narcotic  addicts.  [Sacramento]  Research  Division,  Dept.  of 
Corrections,  State  of  California,  1971.  17  p.  (California,  Dept.  of  Correc- 
tions, Research  report  no.  42)  HV9305.C3A32  no.  42 
Bibliography:  p.  17. 

Braucht,  G.  Nicholas,  and  others.  Drug  education;  a  review  of  goals,  approaches  and 
effectiveness,  and  a  paradigm  for  evaluation.  Quarterly  journal  of  studies  on 
alcohol,  v.  34,  Dec.  1973:  1279-1292.  RC367.A35,  v.  34 

"The  controversy  over  the  goals  and  educational  methods  currently  used  in 
alcohol  and  drug  education  is  outlined.  A  review  of  the  research  on  drug  edu- 
cation programs  indicates  that  there  is  almost  no  empirical  evidence  of  the 
effectiveness  of  these  programs:  strategies  for  future  research  are  outlined." 

Calof,  Judith.  Observations  on  five  residential  facilities  of  the  State  of  New 
York  Narcotic  Addiction  Control  Commission;  a  report  to  the  public.  Prepared 
for  the  Committee  on  Youth  and  Corrections.  New  York,  Dept.  of  Public  Affairs, 
Community  Service  Society  of  New  York,  1971.  103  1. 

HV5831.N7C33 

Costello,  Raymond  M. ,  James  E.  Bechtel ,  and  Martin  B.  Giffen.  A  community's  ef- 
forts to  attack  the  problem  of  alcoholism.  Base  rate  data  for  future  program 
evaluation.  International  journal  of  the  addictions,  v.  8,  1973:  875-888. 

RC566.A1I5,  v.  8 
Outlines  "one  basis  by  which  the  effectiveness  of  the  program  is  to  be 
measured  and  offers  speculative  proposals  for  program  evaluation  by  collating 
follow-up  studies  gathered  from  a  review  of  literature." 

District  of  Columbia.  Dept.  of  Corrections.  Performance  of  corrections  referrals 
under  three  narcotic  addiction  treatment  modalities.  [Washington]  1971. 
36  p.  (Its  Research  report  no.  42)  HV5833.W3A5 

Bibliography:  p.  35-36. 

Jaffe,  Jerome  H.  "We're  building  a  health  system  for  drug  abusers."  Medical  eco- 
nomics, v.  50,  Apr.  2,  1973:  25-28,  35,  37,  41-42,  47. 

R723.5.M4,  v.  50 
Discusses  the  programs  and  evaluation  techniques  developed  by  the  Special 
Action  Office  for  Drug  Abuse  Prevention. 

Kim,  Yong  C.  Alcoholism  treatment  program,  St.  Joseph's  Hospital;  evaluation  report, 
by  Yong  C.  Kim,  with  the  assistance  of  John  Reid.  Regina,  Sask.,  Alcoholism 
Commission  of  Saskatchewan,  1972.  125  1.  RC565.K48 

Bibliography:  p.  117-125. 

Ludwig,  Arnold  M. ,  Jerome  Levine,  and  Louis  H.  Stark.  LSD  and  alcoholism;  a  clini- 
cal study  of  treatment  efficacy.  Springfield,  111.,  Thomas  [1970]  331  p. 

RC565.L8 
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Maidlow,  Spencer  T.,  and  Howard  Berman.  The  economics  of  heroin  treatment.  Ameri- 
can journal  of  public  health,  v.  62,  Oct.  1972:  1397-1406. 

RA421.A41,  v.  62 
Compares  the  costs  of  various  narcotic  rehabilitation  programs  with  the 
costs  to  society  if  an  addict  is  not  cured. 

Miller,  Sheldon,  and  others.  Alcoholism:  a  statewide  program  evaluation.  American 
journal  of  psychiatry,  v.  131,  Feb.  1974:  210-214.       RC321.A52,  v.  131 

Describes  the  method  of  developing  an  evaluation  to  test  the  program 
which  delivers  care  to  alcoholics. 

Murphy,  Brian  C.  A  quantitative  test  of  the  effectiveness  of  an  experimental  treat- 
ment programme  for  delinquent  opiate  addicts.   [Ottawa,  Information  Canada, 
1972]  59  p.  (Research  Centre.  Report  4)  HV5840.C32B75 

National  Research  Council.  Division  of  Behavioral  Sciences.  Report  of  the  Panel 
on  the  Impact  of  Information  on  Drug  Use  and  Misuse;  phase  I.  [Washington] 
National  Academy  of  Sciences  [available  from  NTIS]  1972.  36  p. 

"PB  213  097" 

"The  report  describes  current  drug  information  programs  and  suggests  some 
retrospective  studies  aimed  at  evaluating  their  effect  on  target  populations." 

Stanford  Research  Institute.  Alcoholism  program  monitoring  system  development, 
phase  II.  [Menlo  Park,  Calif.,  distributed  by  NTIS]  1972.  1  v.  (various 
pagings) 

"PB  212  050" 

Presents  the  development  of  a  clinical  data  base  which  can  help  determine 
whether  Alcohol  Treatment  Centers  are  meeting  their  objectives. 

Susman,  Jackwell,  comp.  Drug  use  and  social  policy;  an  AMS  anthology.  New  York, 
AMS  Press  [1972]  595  p.  HV5801.S95 

U.S.  Congress.  Senate.  Committee  on  Armed  Services.  Subcommittee  on  Drug  Abuse 
in  the  Military  Services.  Review  of  military  drug  and  alcohol  programs. 
Hearings,  93d  Congress,  1st  session.  Washington,  U.S.  Govt.  Print.  Off., 
1973.  779  p.  KF26.A746  1973 

Hearings  held  Sept.  18-20,  1973. 

U.S.  General  Accounting  Office.  Limited  use  of  Federal  programs  to  commit  narcotic 
addicts  for  treatment  and  rehabilitation;  report  to  the  Congress  [on  the] 
Department  of  Justice  [and]  Department  of  Health,  Education,  and  Welfare, 
by  the  Comptroller  General  of  the  United  States.  [Washington]  1971.  43  p. 

HV5825.A584  1971 
"B-164031(2),  Sept.  20,  1971" 
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Evaluations  of  the  Medicare  and  Medicaid  Programs 


Berliner,  Howard  S.  The  origins  of  health  insurance  for  the  aged.  International 
journal  of  health  services,  v.  3,  summer  1973:  465-474. 

RA421.I49,  v.  3 
"The  author  presents  political -economic  theory  to  explain  the  origins  and 
actual  operation  of  the  Medicare  program.  The  dynamics  of  this  social  innova- 
tion, as  seen  from  the  political-economic  perspective,  are  then  discussed  and 
their  relevance  to  health  policy  is  shown." 

Butts,  Sarah  A.  Public  Assistance  social  services  related  to  Medicaid.  [Washing- 
ton] U.S.  Social  and  Rehabilitation  Service,  1972.  42  p.  (U.S.  Dept.  of 
Health,  Education,  and  Welfare.  DHEW  publication  no.  (SRS)  72-23011) 

HV687.5.U5B88 
"Presents  the  findings  of  a  study  made  in  three  State  welfare  agencies 
of  the  social  service  activities  of  adult  caseworkers  with  the  Medicaid 
program  and  their  preparation  for  this  responsibility." 

Densen,  Paul  M.  Public  accountability  and  reporting  systems  in  Medicare  and  other 
health  programs.  New  England  journal  of  medicine,  v.  289,  Aug.  23,  1973: 
401-406.  R11.B7,  v.  289 

Addresses  the  questions  of  how  much  and  what  kind  of  information  is 
needed,  how  is  it  to  be  collected,  who  is  responsible  for  the  collection, 
and  how  this  information  can  be  made  available  in  the  most  useful  manner. 

Feingold,  Eugene,  ed.  Medicare:  pclicy  and  politics;  a  case  study  and  policy 
analysis.  San  Francisco,  Chandler  Pub.  Co.  [1966]  317  p. 

HD7102.U4F4 

Goldberg,  George  A.,  Jack  Needleman,  and  Sheridan  L.  Weinstein.  Medical  care 

evaluation  studies:  a  utilization  review  requirement.  JAMA  [Journal  of  the 
American  Medical  Association]  v.  220,  Apr.  17,  1972:  383-387. 

R15.A48,  v.  220 

Granfield,  Michael  E.,  Sheila  L.  Conrad,  and  Marvis  J.  Oehm.  Toward  better  analy- 
sis of  social  programs.  Inquiry,  v.  10,  Dec.  1973:  24-33. 

RA410.A1.I58,  v.  10 
Evaluates  the  impact  and  incidence  of  the  Medi-Cal  program  which  redis- 
tributes incomerin-kind. 

Klarman,  Herbert  E.  Major  public  initiatives  in  health  care.  Public  interest,  no. 
34,  winter  1974:  106-123.  H1.P86,  no.  34 

Reviews  the  government  programs  in  health  care. 
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Mold,  Murray  B.  The  politics  of  evaluation  of  social  programs.  Social  work,  v.  IS 
July  1974:  448-455.  HV1.S644,  v.  19 

"A  brief  analysis  of  the  War  on  Poverty,  Medicare,  and  the  Family  Assist- 
ance plan  shows  that  political  factors  influenced  their  design  and  their  fate 
more  than  need  or  effectiveness." 

New  Jersey.  Legislature.  Office  of  Fiscal  Affairs.  Program  analysis  of  institu- 
tional maintenance  support  payments.  [Trenton]  1974.  3  v. 

Reviews  the  system  by  which  New  Jersey  receives  payment  for  the  indigent 
in  the  state  mental  care  facilities. 

Roghmann,  Klaus  J.,  and  Elbert  A.  Powell.  Impact  of  Medicaid  and  an  0E0  health 
center  on  use  of  dental  services  in  an  urban  area.  Public  health  reports, 
v.  89,  July-Aug.  1974:  325-329.  RA11.B17,  v.  89 

Findings  demonstrate  "that  even  when  financial  barriers  are  removed, 
social  class  stratification  persists  as  an  important  determinant  of  the 
pattern  of  utilization  of  dental  health  service." 

Stuart,  Bruce  C.   Who  gains  from  public  health  programs?  In  The  Nation's  health: 
some  issues.  Philadelphia,  American  Academy  of  Polictical  and  Social  Science, 
1972.   (Its  Annals,  v.  399,  Jan.  1972)  p.  145-150.        H1.A4,  v.  399 
"Analysis  of  the  economic  impacts  arising  from  the  introduction  of 
Medicare  and  Medicaid  into  Federal  and  state  budgets  leads  one  to  question 
the  actual  magnitude  of  gains  which  recipients  can  expect  to  obtain  from  new 
funding  in  the  health  area." 

U.S.  General  Accounting  Office.  Improvements  needed  in  processing  Medicare  claims 
for  physicians'  services  in  Texas:  report  to  the  Congress  [on  the]  Social 
Security  Administration,  Dept.  of  Health,  Education,  and  Welfare,  by  the 
Comptroller  General  of  the  United  States.  [Washington,  1970]  53  p. 

HD7102.U4A54  1970 
"B-164031  (4),  Dec.  31,  1970" 


-  More  needs  to  be  done  to  assure  that  physicians'  services  --  paid  by 
Medicare  and  Medicaid  --  are  necessary,  report  to  the  Congress  [on  the] 
Department  of  Health,  Education,  and  Welfare  by  the  Comptroller  General 
of  the  United  States.   [Washington,  1972]  50  p. 

"B-164031  (4),  Aug.  2,  1972"  HD7102.U4U55  1972 

-  Problems  associated  with  reimbursements  to  hospitals  for  services  furnished 
under  Medicare;  report  to  the  Congress  [on  the]  Social  Security  Administration, 
Department  of  Health,  Education,  and  Welfare  by  the  Comptroller  General  of  the 
United  States.   [Washington,  1972]  65  p. 

"B-164031  (4),  Aug.  3,  1972"  HD7102.U4U55  1972a 
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Evaluations  of  Public  Welfare  Programs; 
Selected  References,  1970-1974 


In  its  first  legislative  platform,  the  newly  formed  American  Public  Welfare 
Association  stated  in  the  November,  19A5,  issue  of  Public  Welfare:   "public  welfare 
represent^  a  basic  guarantee  by  government  of  minimum  standards  of  human  existence 
below  which  none  may  be  permitted  to  fall."  Though  some  have  argued  that  such  stan- 
dards should  exist,  there  have  been  many  and  varied  recommendations  concerning  the 
extent  of  governmental  responsibility  for  welfare  finance  and  administration.   How- 
ever, the  growth  of  public  assistance  programs  since  1945  can  be  easily  recognized. 
Public  welfare  programs  are  today  undergoing  close  scrutiny,  study  and  evaluation 
because  of  their  cost,  size,  and  complexity. 

This  bibliography  cites  published  material  concerned  with  only  one  aspect  of 
the  topic:   the  assessments  that  have  been  made  of  public  welfare  programs  and 
programs  sponsored  by  the  Office  of  Economic  Opportunity  through  the  use  of  evalua- 
tion research  techniques.   The  hallmark  of  such  techniques  is  the  application  of  social 
science  research  for  the  purpose  of  attaining  a  scientific,  objective,  and  comprehensive 
assessment  of  a  program's  effectiveness.   Because  of  its  disciplines  and  purposes, 
evaluation  research  has  a  common  bond  with  other  types  of  social  science  research,  . 
but  the  conditions  under  which  it  operates  provide  it  with  a  character  distinctive 
from  other  methods  of  social  science  research.   The  evaluator  frequently  cannot 
manipulate  the  program  or  control  the  testing  conditions  because  of  the  interactive 
environment  of  an  administrative  process  and/or  its  multiple  goals.   But  through  the 
use  of  computer  simulations,  comparative  studies,  quasi-control  groups,  estimates 
or  other  techniques,  an  evaluator  can  choose  the  most  effective  method  or  combina- 
tion of  methods  to  provide  a  scientific  assessment.   While  it  is  immediately  appar- 
ent that  each  evaluation  must  be  tailored  to  the  particular  needs  of  the  program 
being  studied,  there  are  broad  methodologies  common  to  most  evaluation  research. 


The  following  two-part  bibliography  presents  citations  to  journal  and  mono- 
graphic literature  on  the  broad  methodologies  of  evaluation  research  in  part  one, 
and  in  part  two,  their  specific  application  to  public  welfare  and  0E0  programs.   The 
majority  of  the  citations  are  annotated  and  have  been  chosen  from  the  computerized 
bibliographic  data  base  created  and  maintained  by  the  Library  Services  Division  of 
the  Congressional  Research  Service. 

Another  source  of  information  concerning  the  effectiveness  of  public  welfare 
programs  is  the  reports  of  audits  conducted  and  published  by  the  General  Accounting 
Office.   The  primary  purpose  of  the  audits  is  to  make  independent  examinations  for 
the  Congress  of  the  way  government  agencies  are  discharging  their  financial  respon- 
sibilities.  In  its  audits,  the  General  Accounting  Office  makes  recommendations  to 
effect  greater  economy  and  efficiency  in  agency  operations  and  attempts  to  determine 
whether  alternative  approaches  might  better  accomplish  the  goals  of  the  program. 
This  bibliography  cites  only  a  few  of  the  reports  of  the  Comptroller  General;  others 
may  be  located  in  the  monthly  catalog  of  reports  issued  by  the  General  Accounting 
Office. 

The  material  cited  is  arranged  alphabetically  by  main  entry  within  each  of  the 
two  sections.   Citations  to  items  in  the  classified  collections  of  the  Library  of 
Congress  are  provided  with  call  numbers;  items  designated  Law  are  located  in  the  Law 
Library  and  those  with  Newsp  are  found  in  the  Newspaper  and  Current  Periodical  Room 
of  the  Library  of  Congress.   All  journal  articles  are  provided  with  the  call  number 
for  the  bound  volumes  of  the  journal  regardless  of  whether  the  issues  cited  have 
been  bound  at  this  time. 
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CRS-3 
Evaluation  Research 


Abert,  James  B. ,  and  Murray  Kamrass,  eds.   Social  experiments  and  social  program 

evaluation;  proceedings  of  the  Washington  Operations  Research  Council  symposium. 
Cambridge,  Mass.,  Ballinger  Pub.  Co.  [1974]  199  p.  HN57.A7 

Benefit-cost  analyses  of  Federal  programs;  a  compendium  of  papers  submitted  to  the 
Subcommittee  on  Priorities  and  Economy  in  Government  of  the  Joint  Economic 
Committee,   Congress  of  the  United  States.   Washington,  U.S.  Govt.  Print.  Off., 
1973.   274  p.  HJ7539.B45 

At  head  of  title:   92d  Congress,  2d  session.   Joint  committee  print. 

Partial  contents. — Report  on  a  survey  of  Federal  program  evaluation  prac- 
tices, by  W.V.  Roth,  Jr. — Using  benefit-cost  analysis  to  assess  a  human  re- 
source investment  program,  by  B.F.  Davie. — The  Family  Assistance  Plan:   an 
analysis  and  evaluation,  by  D.L.  Bawden,  G.G.  Cain,  and  L.J.  Hausman. 

Biderman,  Albert  D.,  and  Laure  M.  Sharp.   The  competitive  evaluation  research  indus- 
try.  Washington,  Bureau  of  Social  Science  Research,  1972.   75  p. 

Examines  the  social  program  evaluation  research  "industry"  which  has  re- 
cently appeared  in  the  U.S.  and  the  competitive  aspects  of  that  industry  (in 
bidding  for  contracts,  etc.). 

Buchanan,  Garth,  Pamela  Horst,  and  John  Scanlon.   Improving  Federal  evaluation 
planning.   Evaluation,  v.  1,  no.  2,  1973:   86-90. 

H62.A1E9.V.  1 
Describes  "an  approach  to  federal  evaluation  planning  that  the  Urban 
Institute  has  been  developing  in  work  for  the  Department  of  Labor  and  the 
National  Institute  of  Mental  Health." 

Caro,  Francis  G.,  comp.   Readings  in  evaluation  research.   New  York,  Russell  Sage 
Foundation  [1971]  418  p.  H62.C3425 

Davis,  Howard  R.   Four  ways  to  goal  attainment;  an  overview.   Evaluation,  v.  1,  no. 
2,  1973:   43-48.  H62.A1E9.V.  1 

Outlines  four  different  methods  of  goal  attainment  evaluation. 

Evaluation  research;  strategies  and  methods.   [Pittsburgh]  American  Institutes  for 
Research  [1970]  160  p.  H62.5.U5E9 

Federal  evaluation  policy:   analyzing  the  effects  of  public  programs,  by  Joseph  S. 
Wholey  [and  others].   Washington,  Urban  Institute  [1970]  134  p. 

HN59.F37 
Partial  contents. — Federal  evaluation — what  it  is  and  why  it  is  needed. — 
Administration  of  an  evaluation  system. — Organizational  relationships  and 
responsibilities. — Evaluation  resources. — Methodology. 
Bibliography:   p.  121-134. 

Garn,  Harvey  A.,  and  Mancur  Olson.   Public  services  on  the  assembly  line.   Evalua- 
tion, v.  1,  no. 2,  1973:   36-42,  96.  H62.A1E9.V.  1 
In  companion  articles,  the  authors  expand  on  evaluation  and  the  care  that 
must  be  exercised  when  measuring  public  performance. 
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Gordon,  T.  J.,  S.  Enger,  and  R.  Rochberg.   An  experiment  in  simulation  gaming  for 
social  policy  studies.   Technological  forecasting,  v.  1,  Har.  1970:   241-261. 

T174.I38.V.  1 
"This  paper  describes  the  objectives,  design,  and  play  of  a  simulation 
game  developed  by  the  Institute  for  the  Future  under  a  grant  from  the  Connect- 
icut Research  Commission.   The  general  subject  of  the  research. . .was  the 
design  of  methodologies  which  promised  to  be  useful  in  forecasting  societal 
change  and  the  application  of  these  methodologies  to  derive  societal  forecasts 
for  the  state." 

Guttentag,  Marcia.   Subjectivity  and  its  use  in  evaluation  research.   Evaluation, 

v.  1,  no.  2,  1973:   60-65.  H62.AlE9,v.  1 

Describes  a  model  for  program  evaluation  which  quantifies  a  decision- 
maker's values  on  the  belief  that  the  inherent  subjectivity  of  decision-making 
makes  classical  experimental  designs  not  very  useful. 

Harberger,  Arnold  C.   Project  evaluation;  collected  papers.   Chicago,  Markham  Pub. 
Co.   [1972?]  330  p.  HD47.H28 

Hinrichs,  Harley  H. ,  and  Graeme  M.  Taylor.  Systematic  analysis;  a  primer  on  bene- 
fit-cost analysis  and  program  evaluation.  Pacific  Palisades,  Calif.,  Goodyear 
Pub.  Co.  [1972]  152  p.  HJ2052.H53 

Lyden,  Fremont  James,  and  Lawrence  K.  Lee.   Evaluating  program  change.   Social  work, 
v.  18,  Mar.  1973:   87-94.  HV1.S644.V.  18 

"The  authors  describe  a  methodology  developed  and  used  effectively  in 
one  metropolitan  area  to  measure  program  change.   They  identify  and  utilize 
the  findings  to  show  how  greater  change  might  be  effected  in  more  agencies." 

Meinert,  Roland  G.   Futures  forecasting.   Social  work,  v.  18,  Rbv.  1973:   48-51. 

HV1.S644.V.  18 
"Four  systematic  techniques  of  forecasting  the  future,  widely  practiced 
in  other  fields,  could  be  useful  in  social  planning.   The  techniques  are 
extrapolation,  Delphi  forecasting,  simulation,  and  scenario  speculation." 

Morehouse,  Thomas  A.   The  problem  of  measuring  the  impacts  of  social-action  pro- 
grams.  [Fairbanks,  Institute  of  Social,  Economic  and  Governs 
University  of  Alaska,  1972]  26  p.   (ISECR  occasional  papers,  no.  6) 

H62.5.C5M67 
Bibliography:   p.  24-26. 

Rivlin,  Alice  M.   Systematic  thinking  for  social  action.  Washington,  Brookings 
Institution  [1971]  150  p.   (H.  Rowan  Gaither  lectures  in  systems  science) 

H62.R56 

Rossi,  Peter  Henry,  and  Walter  Williams,  eds.   Evaluating  social  action  programs; 
theory,  practice,  and  politics.   New  York,  Seminar  Press,  1972.   326  p. 

H62.R644 

Suchman,  Edward  Allen.   Evaluation  research;  principles  and  practice  in  public  ser- 
vice &  social  action  programs.   New  York,  Russell  Sage  Foundation,  1967. 
186  p.  H61.S888 
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Symposium  on  policy  analysis.   Urban  affairs  quarterly,  v.  7,  June  1972:   379-412. 

HT101.U67.V.  7 
Contents. — Some  promising  approaches  to  policy  analysis:   an 
introduction,  by  J.  Brandl. — An  advocate's  process  outline  for  policy  analysis: 
the  case  of  welfare  reform,  by  J.  Lyday. — Micro-analytic  simulation  modeling 
for  evaluation  of  public  formulation  and  evaluation,  by  W.  Harrar  and  D. 
Bawden. — The  capacity  of  social  science  organizations  to  perform  large-scale 
evaluative  research,  by  W.  Williams. 

Wegner,  Robert  E.  C.   Evaluating  the  evaluators.   Public  administration  review, 

v.  33,  Jan. -Feb.  1973:   80-87.  JKl.P85,v.  33 

Weiss,  Carol  H.   Abstracts  on  evaluative  research,  by  Carol  H.  Weiss  [and  others, 
n.p.]  Model  Cities  Evaluation  Institute  [1973?]  1  v.   (unpaged) 

Extensive  abstracts  of  articles  and  books  on  evaluation  in  a  variety  of 
program  areas  are  arranged  alphabetically  by  author  and  a  subject /author  index 
is  provided. 

Between  the  cup  and  the  lip...   Evaluation,  v.  1,  no.  2,  1973:   49-55. 

H62.A1E9.V.  1 
"That  much  evaluation  research  falls  short  of  the  expectations  of  funders, 
program  staff,  and  evaluators  themselves  is  due  less  to  limitations  of  research 
expertise  than  to  organizational  constraints  on  the  researcher's  ability  to 
apply  what  he  knows." 


Evaluative  research:  methods  for  assessing  program  effectiveness.   Englewood 

Cliffs,  N.J.,  Prentice-Hall  [1972]  160  p.  H62.W3962 

Wholey,  Joseph  S.   What  can  we  actually  get  from  program  evaluation?  Policy 

sciences,  v.  3,  Sept.  1972:   361-369.  Hl.P7,v.  3 

"Assesses  the  role  program  evaluation  can  play  in  assisting  decisions  on 
public  programs." 

Wilemon,  David.   Transferring  space-age  management  technology.   Conference  Board 

record,  v.  7,  Oct.  1970:   50-55.  HC101.C64,v.  7 

"This  article  explores  the  characteristics  of  project  management  to 
appraise  its  transfer  potentialities  in  managing  society's  complex  problems, 
first  examining:. . . some  of  the  generalized  characteristics  of  these  "new 
management  techniques." 

Williams,  Walter.   The  capacity  of  social  science  organizations  to  perform  large- 
scale  evaluative  research.   Seattle  [Institute  of  Governmental  Research, 
University  of  Washington]  1971.   47  p.   (Public  policy  paper,  no.  2) 

JA37.P9  no.  2 


Evaluations  of  Public  Welfare  and  0E0  Programs 


Abels,  Paul.    The  managers  are  coming!   the  managers  are  coming!   Public  welfare, 
v.  31,  fall  1973:   13-15.  Hl.P75,v.  31 

Subjects  the  public  welfare  hierarchy  to  systems  analysis  and  finds  that 
it  is  a  nonsystem. 

Allardt,  Erik.    A  welfare  model  for  selecting  indicators  of  national  development. 
Policy  sciences,  v.  4,  Mar.  1973:   63-74.  Hl.P7,v.  4 

Baldwin,  Fred  D.    Evaluating  the  0E0  Legal  Services  Program.   Policy  sciences,  no. 
4,  Sept.  1973:   347-364.  H1.P7,  no.  4 

Reports  on  an  evaluation  of  the  0E0  Legal  Services  Program  which  includes 
an  explanation  of  how  the  evaluation  was  performed,  some  of  the  findings,  and 
why  0E0  modified  or  abandoned  some  of  the  objectives  of  the  study  before  its 
completion. 

Benton,  William  B. ,  Jr.  Maryland  management  uses  QC  findings.  Social  and  rehabili- 
tation record,  v.  1,  July-Aug.  1974:   9-13.  HD7256.U5S62,v.  1 

Outlines  the  procedures  that  have  been  instituted  in  Maryland  to  provide 
for  quality  control  of  the  AFDC  program. 

Boettcher,  Richard  E.    A  test  of  Etzioni's  compliance  theory  in  a  welfare  system. 
Public  welfare,  v.  31,  summer  1973:   43-46,  48-49.  HVl.P75,v.  31 

Brookings  Institution,  Washington,  D.  C.    Functions  and  activities  of  the  National 
Government  in  the  field  of  welfare;  a  report  with  recommendations.   Prepared 
for  the  Commission  on  Organization  of  the  Executive  Branch  of  the  Government. 
New  York,  Greenwood  Press  [1969]   589  p.  HV85.A52  1969 

Buttrick,  Shirley  M.    Present  shock:   the  future  of  the  social  services.   Public 
welfare,  v.  31,  fall  1973:   41-42,  44-46.  HV1.P75.V.  31 

Says  that  a  review  of  the  trends  in  social  work  suggests  a  need  for  a 
definition  of  priorities,  development  of  working  relationships  with  the  busi- 
ness or  private  sector,  and  an  expanded  concept  of  welfare  planning  by  the 
Federal  Government. 

Capoccia,  Victor  A.   Social  welfare  planning  and  the  new  Federalism:   the  Allied 

Services  Act.   Journal  of  the  American  Institute  of  Planners,  v.  39,  July  1973: 
244-253.  NA9000.A45,v.  39 

"As  it  now  stands,  the  legislation  has  more  potential  for  improving  the 
management  of  services  in  a  narrow  sense  of  the  concept  than  for  any  signifi- 
cant plannine  for  social  welfare  services  on  a  svstemwide  basis." 

Carroll,  Stephen  J.,  David  H.  Greenberg,  and  Patricia  0.  Katsky.  Evaluation  results 
for  the  Los  Angeles  Women's  Job  Corps  Center.  Santa  Monica,  Calif.,  Rand,  1971 
72  p.   ([Rand  Corporation.   Rand  report]  R-745-0E0)  AS36.R3  R-745 
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Champagne,  Anthony.   An  evaluation  of  the  effectiveness  of  the  0E0  Legal  Services 
Program.   Urban  affairs  quarterly,  v.  9,  June  1974:   466-513. 

HT101.U67.V.  9 

Evaluation  "clearly  shows  that  the  Legal  Services  Program  has  been  quite 
successful  in  meeting  its  goals  and,  therefore,  in  providing  for  the  legal 
needs  of  poor  people.   Unfortunately ,  the  success  of  the  program  has  caused  a 
number  of  political  leaders  to  concern  themselves  with  the  program — a  concern 
aimed,  oddly  enough,  not  at  improving  what  this  research  shows  as  being  an 
already  successful  program,  but  in  reducing  the  program's  effectiveness." 

Champagne's  article  is  followed  by  a  criticism  written  by  Richard  A.  Beck. 

The  internal  operation  of  0E0  legal  services  projects.   Journal  of  urban  law, 

v.  51,  May  1974:   649-685.  Law 

Article  uses  organization  theory  to  "make  a  preliminary  examination  of 
legal  services  project  internal  operations  by  examining:   (1)  legal  services 
project  directors;  (2)  legal  services  staff  attorneys;  and  (3)  the  local 
project  governing  board." 

Chommie,  Peter  W. ,  and  Joe  Hudson.   Evaluation  of  outcome  and  process.   Social 

work,  v.  19,  Nov.  1974:   682-687.  HVl.S644,v.  19 

"Evaluating  program  outcome  provides  the  verification  that  planners  and 
policy-makers  need  to  decide  a  program's  future.   Evaluating  program  process 
leads  to  the  discovery  of  facts  that  explain  outcomes,  make  goals  specific, 
and  improve  delivery  of  services." 

Citizens'  Board  of  Inquiry  into  Hunger  and  Malnutrition  in  the  United  States. 
Hunger,  U.S.A.;  a  report.   Washington,  Sew  Community  Press,  1968.   100  p. 

HD9005.C57 

Committee  for  Economic  Development.   Improving  the  public  welfare  system:   a  state- 
ment by  the  Research  and  Policy  Committee.   New  York,  1970.   75  p. 

HV95.C62 

Committee  for  Economic  Development.   Research,  and  Policy  Committee.    Improving 

Federal  program  performance;  a  statement  on  national  policy.   [New  York]  1971. 
86  p.  HJ7539.C65 

Contents. — Introduction  and  summary  of  recommendations. — Setting  objec- 
tives: the  basis  for  program  design  and  management. — Developing  programs  to 
achieve  objectives. — Program  execution  and  performance  evaluation. 

Conly,  Sonia.   Success  and  failure  in  the  work  incentive  program.   Growth  and 

change,  v.  4,  Oct.  1973:   16-23.  HT390.G74,v.  4 

Presents  pilot  cost-benefit  study  of  the  WIN  program  for  welfare  mothers 
in  Florence,  South  Carolina. 

Costello,  Joan.   Review  and  summary  of  A  National  Survey  of  the  Parent-child 

Center  Program  by  Joan  Costello,  with  Eleanor  Binstock.   [Washington,  U.  S. 
Bureau  of  Head 'Start  and  Early  Childhood,  1971]   47  p. 

HV741.C66 
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Cox,  Irene.   Handbook  of  public  income  transfer  programs;  a  staff  study  prepared 
for  the  use  of  the  Subcommittee  on  Fiscal  Policy  of  the  Joint  Economic 
Committee,  Congress  of  the  United  States.   Washington,  U.S.  Govt.  Print.  Off., 
1972.   336  p.   (Studies  in  public  welfare,  paper  no.  2)       HV91.C68 
At  head  of  title:   92d  Congress,  2d  session.   Joint  committee  print. 

Bibliography:   p.  334-336. 

Davidson,  Roger  H. ,  and  Sar  A.  Levitan.   Antipoverty  housekeeping;  the  administra- 
tion of  the  Economic  Opportunity  Act.   Ann  Arbor,  Institute  of  Labor  and 
Industrial  Relations  (University  of  Michigan  -  Wayne  State  University)   1968. 
83  p.   (Policy  papers  in  human  resources  and  industrial  relations,  no.  9) 

HC110.P63D3 

Evaluating  the  war  on  poverty.   Special  editor  of  this  volume,  Louis  A.  Ferman. 

Philadelphia,  American  Academy  of  Political  and  Social  Science,  1969.   251  p. 
(Annals,  v.  385,  Sept.  1969)  Hl.A4,v.  385 

Partial  contents. — The  war  on  poverty:   experiment  in  Federalism,  by 
R.H.  Davidson. — The  politics  of  evaluation:   the  case  of  Head  Start,  by  W. 
Williams  and  J.W.  Evans. — The  evaluation  of  broad-aim  programs:   a  cautionary 
case  and  a  moral,  by  R.  S.  Weiss  and  M.  Rein. — Some  perspectives  on  evaluating 
social  welfare  programs,  by  L.  A.  Ferman. 

Feldman,  Frances  Lomas.   Human  services  in  rural  Alaska:   highlights  from  the 
evaluation  of  the  Rural  Areas  Social  Services  Project.   [n.p.,  1971?] 
214  p.  HV98.A4F39 


Organization  and  delivery  of  human  services  in  rural  Alaska:   evaluation  of 

the  Rural  Areas  Social  Services  Project.   [Los  Angeles,  University  of  Southern 
California,  Regional  Research  Institute  in  Social  Welfare,  1971]   1  v. 
(various  pagings)  HV98.A4F4 

Reaching  rural  Alaskan  natives  through  human  services  aides.   Welfare  in 

review,  v.  9,  May-June  1971:   9-14.  HV1.W36.V.  9 

This  article  is  based  on  a  comprehensive  report  of  the  evaluative 
research  of  a  rural  Alaskan  demonstration  project  (Rural  Areas  Social  Services 
Project)  supported  under  sec.  1115  of  the  Social  Security  Act  by  the  Social 
and  Rehabilitation  Service  of  HEW. 

Fischer,  Joel.    Is  casework  effective?;  a  review.   Social  work,  v.  18,  Jan.  1973: 
5-20.  HVl.S644,v.  18 

Review  of  controlled  research  found  caseworker  ineffectiveness  to  be  the 
rule  rather  than  the  exception. 

Galm,  Sharon.   Welfare  -  an  administrative  nightmare.   A  staff  study  prepared  for 
the  use  of  the  Subcommittee  on  Fiscal  Policy  of  the  Joint  Economic  Committee, 
Congress  of  the  United  States.   Washington,  U.S.  Govt.  Print.  Off.,  1972. 
44  p.    (Issues  in  welfare  administration,  pt.  1)   (Studies  in  public  welfare, 
paper  no.  5,  pt.  1)  HV95.I88  pt.  1 

At  head  of  title:   92d  Congress,  2d  session.   Joint  committee  print. 
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Garfinkel,  Irwin,  and  Larry  L.  Orr.  Welfare  policy  and  the  employment  rate  of  the 
AFDC  mothers.   National  tax  journal,  v.  27,  June  1974:   275-284. 

HJ2240.N3135,v.  27 
"The  paper  analyzes  the  effect  of  economic  incentives  in  the  AFDC  program 
on  the  employment  of  AFDC  mothers.   Short  of  a  drastic  reduction  in  guarantees, 
however,  manipulation  of  any  of  these  program  parameters  is  not  likely  to  in- 
duce most  AFDC  mothers  to  work." 
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— The  AFDC  numbers  game,  by  B.  Raskin. — Poverty  and  poor  health,  by  D.  Swanston. 
— You  are  what  you  eat:   feeding  D.  C.'s  hungry,  by  M.  L.  and  N.  Kotz. — The  pay- 
off of  the  Federal  social  dollar,  by  S.  A.  Levitan. 

Lyden,  Fremont  James,  and  Lawrence  K.  Lee.   Evaluating  program  change.   Social  Work, 
v.  18,  Mar.  1973:  87-94.  HVl.S644,v.  18 

"The  authors  describe  a  methodology  developed  and  used  effectively  in  one 
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